Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[wasm][AOT] emcc : error - received SIGKILL (-9) #89402

Open
pavelsavara opened this issue Jul 24, 2023 · 9 comments
Open

[wasm][AOT] emcc : error - received SIGKILL (-9) #89402

pavelsavara opened this issue Jul 24, 2023 · 9 comments
Assignees
@radekdoulik
Labels
arch-wasm WebAssembly architecture area-Codegen-AOT-mono Known Build Error Use this to report build issues in the .NET Helix tab os-browser Browser variant of arch-wasm wasm-aot-test WebAssembly AOT Test
Milestone
10.0.0

Comments

@pavelsavara
Copy link
Member

pavelsavara commented Jul 24, 2023 •
edited by build-analysis bot
Loading

Build: https://dev.azure.com/dnceng-public/public/_build/results?buildId=653707

Error Blob

{
  "BuildRetry": false,
  "ErrorPattern": ["emcc", "SIGKILL"],
  "ExcludeConsoleLog": false
}

Reproduction Steps

AOT wasm Microsoft.Extensions.Options.SourceGeneration.Unit.Tests

Log

Known issue validation

Build: 🔎 https://dev.azure.com/dnceng-public/public/_build/results?buildId=653707
Error message validated: [emcc SIGKILL]
Result validation: ✅ Known issue matched with the provided build.
Validation performed at: 4/25/2024 7:29:32 AM UTC

Report

Build Definition Test Pull Request
973172 dotnet/runtime WasmTestOnChrome-ST-System.Runtime.Tests.WorkItemExecution
973163 dotnet/runtime WasmTestOnChrome-ST-System.Runtime.Tests.WorkItemExecution
973064 dotnet/runtime WasmTestOnChrome-ST-System.Runtime.Tests.WorkItemExecution #113297
973047 dotnet/runtime WasmTestOnChrome-ST-System.Runtime.Tests.WorkItemExecution #113041
973032 dotnet/runtime WasmTestOnChrome-ST-Microsoft.Extensions.Options.SourceGeneration.Unit.Tests.WorkItemExecution
972999 dotnet/runtime WasmTestOnChrome-ST-System.Runtime.Tests.WorkItemExecution #113041
972982 dotnet/runtime WasmTestOnChrome-ST-System.Runtime.Tests.WorkItemExecution #113048
972861 dotnet/runtime WasmTestOnChrome-ST-System.Runtime.Tests.WorkItemExecution
972843 dotnet/runtime normal-Microsoft.Extensions.Options.SourceGeneration.Unit.Tests.WorkItemExecution
972783 dotnet/runtime WasmTestOnChrome-ST-System.Runtime.Tests.WorkItemExecution #112937
972793 dotnet/runtime WasmTestOnChrome-ST-System.Runtime.Tests.WorkItemExecution #113288
972650 dotnet/runtime WasmTestOnChrome-ST-System.Runtime.Tests.WorkItemExecution #113288
972641 dotnet/runtime WasmTestOnChrome-ST-System.Runtime.Tests.WorkItemExecution #113287
972624 dotnet/runtime WasmTestOnChrome-ST-System.Runtime.Tests.WorkItemExecution #113243
972473 dotnet/runtime WasmTestOnChrome-ST-System.Runtime.Tests.WorkItemExecution #113191
972470 dotnet/runtime WasmTestOnChrome-ST-System.Runtime.Tests.WorkItemExecution #113056
972456 dotnet/runtime WasmTestOnChrome-ST-System.Runtime.Tests.WorkItemExecution #113284
972438 dotnet/runtime WasmTestOnChrome-ST-System.Runtime.Tests.WorkItemExecution #113280
972378 dotnet/runtime WasmTestOnChrome-ST-System.Runtime.Tests.WorkItemExecution #113277
972350 dotnet/runtime WasmTestOnChrome-ST-System.Runtime.Tests.WorkItemExecution #113243
972304 dotnet/runtime WasmTestOnChrome-ST-System.Runtime.Tests.WorkItemExecution #112378
972269 dotnet/runtime WasmTestOnChrome-ST-System.Private.Xml.Tests.WorkItemExecution
972274 dotnet/runtime WasmTestOnChrome-ST-System.Runtime.Tests.WorkItemExecution #113248
972197 dotnet/runtime normal-Microsoft.Extensions.Options.SourceGeneration.Unit.Tests.WorkItemExecution
972222 dotnet/runtime WasmTestOnChrome-ST-System.Runtime.Tests.WorkItemExecution #112997
972238 dotnet/runtime WasmTestOnChrome-ST-System.Runtime.Tests.WorkItemExecution
972176 dotnet/runtime WasmTestOnChrome-ST-System.Runtime.Tests.WorkItemExecution #113137
972152 dotnet/runtime WasmTestOnChrome-ST-System.Runtime.Tests.WorkItemExecution #111934
972130 dotnet/runtime WasmTestOnChrome-ST-System.Runtime.Tests.WorkItemExecution #113267
972113 dotnet/runtime WasmTestOnChrome-ST-System.Runtime.Tests.WorkItemExecution #113257
969782 dotnet/runtime WasmTestOnChrome-ST-System.Runtime.Tests.WorkItemExecution #113166
972051 dotnet/runtime normal-Microsoft.Extensions.Options.SourceGeneration.Unit.Tests.WorkItemExecution
972069 dotnet/runtime WasmTestOnChrome-ST-System.Runtime.Tests.WorkItemExecution #111680
971989 dotnet/runtime WasmTestOnChrome-ST-System.Runtime.Tests.WorkItemExecution #113041
971931 dotnet/runtime WasmTestOnChrome-ST-System.Runtime.Tests.WorkItemExecution #113258
971764 dotnet/runtime WasmTestOnChrome-ST-System.Runtime.Tests.WorkItemExecution #112178
971748 dotnet/runtime WasmTestOnChrome-ST-System.Runtime.Tests.WorkItemExecution #112876
971689 dotnet/runtime WasmTestOnChrome-ST-System.Runtime.Tests.WorkItemExecution #111680
971666 dotnet/runtime WasmTestOnChrome-ST-System.Runtime.Tests.WorkItemExecution #113014
971661 dotnet/runtime WasmTestOnChrome-ST-System.Runtime.Tests.WorkItemExecution
971632 dotnet/runtime WasmTestOnChrome-ST-System.Runtime.Tests.WorkItemExecution
971543 dotnet/runtime WasmTestOnChrome-ST-System.Runtime.Tests.WorkItemExecution #113241
971589 dotnet/runtime WasmTestOnChrome-ST-System.Runtime.Tests.WorkItemExecution #113248
971477 dotnet/runtime WasmTestOnChrome-ST-System.Runtime.Tests.WorkItemExecution #113241
971399 dotnet/runtime WasmTestOnChrome-ST-System.Runtime.Tests.WorkItemExecution #111934
971337 dotnet/runtime WasmTestOnChrome-ST-System.Runtime.Tests.WorkItemExecution #113184
971125 dotnet/runtime WasmTestOnChrome-ST-System.Runtime.Tests.WorkItemExecution #113095
971274 dotnet/runtime WasmTestOnChrome-ST-System.Runtime.Tests.WorkItemExecution #112417
971277 dotnet/runtime WasmTestOnChrome-ST-System.Runtime.Tests.WorkItemExecution #112417
971221 dotnet/runtime WasmTestOnChrome-ST-System.Runtime.Tests.WorkItemExecution #112997
971164 dotnet/runtime WasmTestOnChrome-ST-System.Runtime.Tests.WorkItemExecution #113232
971108 dotnet/runtime WasmTestOnChrome-ST-System.Private.Xml.Tests.WorkItemExecution
971094 dotnet/runtime WasmTestOnChrome-ST-System.Runtime.Tests.WorkItemExecution #107230
971051 dotnet/runtime WasmTestOnChrome-ST-System.Runtime.Tests.WorkItemExecution
971042 dotnet/runtime WasmTestOnChrome-ST-System.Runtime.Tests.WorkItemExecution #113229
970842 dotnet/runtime WasmTestOnChrome-ST-System.Runtime.Tests.WorkItemExecution #113014
969632 dotnet/runtime WasmTestOnChrome-ST-System.Runtime.Tests.WorkItemExecution #112616
970733 dotnet/runtime WasmTestOnChrome-ST-System.Runtime.Tests.WorkItemExecution #113165
970570 dotnet/runtime WasmTestOnChrome-ST-System.Runtime.Tests.WorkItemExecution #113207
970713 dotnet/runtime WasmTestOnChrome-ST-System.Runtime.Tests.WorkItemExecution #113164
970653 dotnet/runtime WasmTestOnChrome-ST-System.Runtime.Tests.WorkItemExecution #108365
970649 dotnet/runtime WasmTestOnChrome-ST-System.Runtime.Tests.WorkItemExecution #112719
970638 dotnet/runtime WasmTestOnChrome-ST-System.Runtime.Tests.WorkItemExecution #113212
970605 dotnet/runtime WasmTestOnChrome-ST-System.Runtime.Tests.WorkItemExecution #113211
967751 dotnet/runtime WasmTestOnChrome-ST-System.Runtime.Tests.WorkItemExecution #113014
970574 dotnet/runtime WasmTestOnChrome-ST-System.Runtime.Tests.WorkItemExecution #113209
970517 dotnet/runtime WasmTestOnChrome-ST-System.Runtime.Tests.WorkItemExecution
970536 dotnet/runtime WasmTestOnChrome-ST-System.Runtime.Tests.WorkItemExecution
970330 dotnet/runtime WasmTestOnChrome-ST-System.Runtime.Tests.WorkItemExecution #113146
970499 dotnet/runtime WasmTestOnChrome-ST-System.Runtime.Tests.WorkItemExecution #113164
970404 dotnet/runtime WasmTestOnChrome-ST-System.Runtime.Tests.WorkItemExecution #112937
970397 dotnet/runtime WasmTestOnChrome-ST-System.Runtime.Tests.WorkItemExecution #113190
970321 dotnet/runtime WasmTestOnChrome-ST-System.Runtime.Tests.WorkItemExecution #107230
970230 dotnet/runtime WasmTestOnChrome-ST-System.Runtime.Tests.WorkItemExecution #113185
970222 dotnet/runtime WasmTestOnChrome-ST-System.Runtime.Tests.WorkItemExecution #113111
970216 dotnet/runtime WasmTestOnChrome-ST-System.Runtime.Tests.WorkItemExecution #113056
969731 dotnet/runtime WasmTestOnChrome-ST-System.Runtime.Tests.WorkItemExecution #113140
970172 dotnet/runtime WasmTestOnChrome-ST-System.Runtime.Tests.WorkItemExecution #112891
970126 dotnet/runtime WasmTestOnChrome-ST-System.Runtime.Tests.WorkItemExecution #113167
970136 dotnet/runtime WasmTestOnChrome-ST-System.Runtime.Tests.WorkItemExecution #113146
970062 dotnet/runtime WasmTestOnChrome-ST-System.Runtime.Tests.WorkItemExecution #113175
969858 dotnet/runtime WasmTestOnChrome-ST-System.Runtime.Tests.WorkItemExecution #113169
970059 dotnet/runtime WasmTestOnChrome-ST-System.Runtime.Tests.WorkItemExecution #113027
969854 dotnet/runtime normal-Microsoft.Extensions.Options.SourceGeneration.Unit.Tests.WorkItemExecution
970033 dotnet/runtime WasmTestOnChrome-ST-System.Runtime.Tests.WorkItemExecution #113171
969995 dotnet/runtime WasmTestOnChrome-ST-System.Private.Xml.Tests.WorkItemExecution
969964 dotnet/runtime WasmTestOnChrome-ST-System.Runtime.Tests.WorkItemExecution
969955 dotnet/runtime WasmTestOnChrome-ST-System.Runtime.Tests.WorkItemExecution #112548
969948 dotnet/runtime WasmTestOnChrome-ST-System.Runtime.Tests.WorkItemExecution #113167
969882 dotnet/runtime WasmTestOnChrome-ST-System.Runtime.Tests.WorkItemExecution #113137
969167 dotnet/runtime WasmTestOnChrome-ST-System.Runtime.Tests.WorkItemExecution #112831
969776 dotnet/runtime WasmTestOnChrome-ST-System.Runtime.Tests.WorkItemExecution #113165
969287 dotnet/runtime WasmTestOnChrome-ST-System.Runtime.Tests.WorkItemExecution #100729
969532 dotnet/runtime WasmTestOnChrome-ST-System.Runtime.Tests.WorkItemExecution #112719
969211 dotnet/runtime WasmTestOnChrome-ST-System.Runtime.Tests.WorkItemExecution #108365
969403 dotnet/runtime WasmTestOnChrome-ST-System.Runtime.Tests.WorkItemExecution #111050
969384 dotnet/runtime WasmTestOnChrome-ST-System.Runtime.Tests.WorkItemExecution
969388 dotnet/runtime WasmTestOnChrome-ST-System.Runtime.Tests.WorkItemExecution #113124
969362 dotnet/runtime WasmTestOnChrome-ST-System.Runtime.Tests.WorkItemExecution
969333 dotnet/runtime WasmTestOnChrome-ST-System.Runtime.Tests.WorkItemExecution #111610
Displaying 100 of 550 results

Summary

24-Hour Hit Count 7-Day Hit Count 1-Month Count
7 158 545
@pavelsavara pavelsavara added arch-wasm WebAssembly architecture blocking-clean-ci Blocking PR or rolling runs of 'runtime' or 'runtime-extra-platforms' area-Codegen-AOT-mono wasm-aot-test WebAssembly AOT Test Known Build Error Use this to report build issues in the .NET Helix tab os-browser Browser variant of arch-wasm labels Jul 24, 2023
@pavelsavara pavelsavara added this to the 8.0.0 milestone Jul 24, 2023
@ghost
Copy link

ghost commented Jul 24, 2023

Tagging subscribers to 'arch-wasm': @lewing
See info in area-owners.md if you want to be subscribed.

Issue Details

Error Blob

{
  "BuildRetry": false,
  "ErrorPattern": "emcc.*SIGKILL",
  "ExcludeConsoleLog": true
}

Reproduction Steps

AOT wasm Microsoft.Extensions.Options.SourceGeneration.Unit.Tests

Log

Author: pavelsavara
Assignees: -
Labels:

arch-wasm, blocking-clean-ci, area-Codegen-AOT-mono, wasm-aot-test, Known Build Error, os-browser
Milestone: 8.0.0

@pavelsavara pavelsavara mentioned this issue Jul 24, 2023
Merged
@lewing
Copy link
Member

lewing commented Jul 24, 2023

Is this it hitting the ulimit of the container?

@lewing lewing self-assigned this Jul 24, 2023
@radical
Copy link
Member

radical commented Jul 24, 2023 

[ 7455.672962] [ 2825] 1000 2825 158937 41558 684032 0 0 python3 
[ 7455.672964] [ 4160] 102 4160 67335 453 180224 0 0 rsyslogd 
[ 7455.672966] [ 4181] 0 4181 455864 16951 634880 0 0 amacoreagent 
[ 7455.672968] [ 4453] 0 4453 298556 3977 249856 0 0 auoms 
[ 7455.672969] [ 4495] 0 4495 215902 6240 204800 0 0 auomscollect 
[ 7455.672971] [ 4750] 102 4750 191707 14176 622592 0 0 mdsd 
[ 7455.672973] [ 4967] 0 4967 334103 3030 253952 0 0 azsecd 
[ 7455.672975] [ 5046] 0 5046 352600 4193 270336 0 0 azsecmond 
[ 7455.672977] [ 32457] 0 32457 317321 3577 258048 0 0 azsecd 
[ 7455.672979] [ 2226] 0 2226 376486 334278 2924544 0 0 clamscan 
[ 7455.672981] [ 22067] 0 22067 180189 824 114688 0 -998 containerd-shim 
[ 7455.672983] [ 22089] 1000 22089 1159 16 57344 0 0 helix_docker_wo 
[ 7455.672985] [ 22156] 1000 22156 1159 17 53248 0 0 execute.sh 
[ 7455.672987] [ 22212] 1000 22212 5047 87 81920 0 0 bash 
[ 7455.672989] [ 22215] 1000 22215 68517493 67031 1163264 0 0 dotnet 
[ 7455.672990] [ 23320] 1000 23320 1159 17 61440 0 0 sh 
[ 7455.672992] [ 23321] 1000 23321 34332 22818 335872 0 0 python3 
[ 7455.672994] [ 23345] 1000 23345 1420573 1386396 11264000 0 0 wasm-opt 
[ 7455.672995] oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=/,mems_allowed=0,global_oom,task_memcg=/docker/a2a4080691301bb74e3b282eab0b6f638e9d6f1227e62967b7fd46654706e554,task=wasm-opt,pid=23345,uid=1000 
[ 7455.673018] Out of memory: Killed process 23345 (wasm-opt) total-vm:5682292kB, anon-rss:5545584kB, file-rss:0kB, shmem-rss:0kB, UID:1000 pgtables:11000kB oom_score_adj:0

@lewing
Copy link
Member

lewing commented Jul 25, 2023

lets disable these tests in aot for now

@lewing lewing mentioned this issue Aug 13, 2023
Open
3 tasks
@lewing
Copy link
Member

lewing commented Aug 13, 2023

closting as a duplicate of dotnet/dnceng#450

@lewing lewing closed this as completed Aug 13, 2023
@ghost ghost locked as resolved and limited conversation to collaborators Sep 12, 2023
@pavelsavara
Copy link
Member Author

I see this again https://helixre107v0xdcypoyl9e7f.blob.core.windows.net/dotnet-runtime-refs-pull-99833-merge-af0760c542d34f95a8/normal-System.Runtime.Tests/1/console.f849dad3.log?helixlogtype=result

@pavelsavara pavelsavara reopened this Apr 2, 2024
@lewing
Copy link
Member

lewing commented Apr 24, 2024

Analysis claims there are recent failures but the links are not updated?

@pavelsavara
Copy link
Member Author

we run most of those problematic cases only in runtime-wasm pipeline. Which needs to be triggered manually.
I updated the filter

@lewing lewing removed the blocking-clean-ci Blocking PR or rolling runs of 'runtime' or 'runtime-extra-platforms' label May 2, 2024
@dotnet dotnet unlocked this conversation May 27, 2024
@akoeplinger akoeplinger modified the milestones: 8.0.0, 9.0.0 May 27, 2024
@lewing lewing modified the milestones: 9.0.0, 10.0.0 Aug 15, 2024
@lewing
Copy link
Member

lewing commented Feb 12, 2025

all the recent failures I looked at are in wasm-opt which is known to have pretty degenerate memory behavior when linking with debug symbols. I think we should investigate what we can do here. @radekdoulik lets discuss offline

@lewing lewing assigned radekdoulik and unassigned lewing Feb 12, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@radekdoulik radekdoulik

Labels
arch-wasm WebAssembly architecture area-Codegen-AOT-mono Known Build Error Use this to report build issues in the .NET Helix tab os-browser Browser variant of arch-wasm wasm-aot-test WebAssembly AOT Test
Projects
None yet
Milestone
10.0.0
Development

No branches or pull requests
5 participants
@radical @lewing @pavelsavara @akoeplinger @radekdoulik