Skip to content

[wasm][AOT] emcc : error - received SIGKILL (-9) #89402

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
pavelsavara opened this issue Jul 24, 2023 · 12 comments
Open

[wasm][AOT] emcc : error - received SIGKILL (-9) #89402

pavelsavara opened this issue Jul 24, 2023 · 12 comments
Assignees
@radekdoulik
Labels
arch-wasm WebAssembly architecture area-Codegen-AOT-mono Known Build Error Use this to report build issues in the .NET Helix tab os-browser Browser variant of arch-wasm wasm-aot-test WebAssembly AOT Test
Milestone
10.0.0

Comments

@pavelsavara
Copy link
Member

pavelsavara commented Jul 24, 2023 •
edited by build-analysis bot
Loading

Build: https://dev.azure.com/dnceng-public/public/_build/results?buildId=653707

Error Blob

{
  "BuildRetry": false,
  "ErrorPattern": ["emcc", "SIGKILL"],
  "ExcludeConsoleLog": false
}

Reproduction Steps

AOT wasm Microsoft.Extensions.Options.SourceGeneration.Unit.Tests

Log

Known issue validation

Build: 🔎 https://dev.azure.com/dnceng-public/public/_build/results?buildId=653707
Error message validated: [emcc SIGKILL]
Result validation: ✅ Known issue matched with the provided build.
Validation performed at: 4/25/2024 7:29:32 AM UTC

Report

Build Definition Test Pull Request
1047421 dotnet/runtime normal-Microsoft.Extensions.Options.SourceGeneration.Unit.Tests.WorkItemExecution
1040556 dotnet/runtime WasmTestOnChrome-ST-System.Runtime.Tests.WorkItemExecution #115518
1043707 dotnet/runtime WasmTestOnChrome-ST-System.Runtime.Tests.WorkItemExecution #115659
1043674 dotnet/runtime WasmTestOnChrome-ST-System.Runtime.Tests.WorkItemExecution #115657
1043660 dotnet/runtime WasmTestOnChrome-ST-System.Runtime.Tests.WorkItemExecution #115656
1043654 dotnet/runtime WasmTestOnChrome-ST-System.Runtime.Tests.WorkItemExecution #115653
1043626 dotnet/runtime WasmTestOnChrome-ST-System.Runtime.Tests.WorkItemExecution #115408
1043422 dotnet/runtime WasmTestOnChrome-ST-System.Threading.Tasks.Dataflow.Tests.WorkItemExecution
1043408 dotnet/runtime WasmTestOnChrome-ST-System.Runtime.Tests.WorkItemExecution #115531
1043376 dotnet/runtime WasmTestOnV8-ST-System.Runtime.Tests.WorkItemExecution #115576
1043374 dotnet/runtime normal-Microsoft.Extensions.Options.SourceGeneration.Unit.Tests.WorkItemExecution
1043341 dotnet/runtime WasmTestOnChrome-ST-System.Runtime.Tests.WorkItemExecution #115366
1043242 dotnet/runtime WasmTestOnChrome-ST-System.Runtime.Tests.WorkItemExecution #115408
1043112 dotnet/runtime WasmTestOnChrome-ST-System.Runtime.Tests.WorkItemExecution #115590
1043079 dotnet/runtime WasmTestOnChrome-ST-System.Runtime.Tests.WorkItemExecution #113956
1043041 dotnet/runtime WasmTestOnChrome-ST-System.Runtime.Tests.WorkItemExecution #115131
1043008 dotnet/runtime WasmTestOnChrome-ST-System.Runtime.Intrinsics.Tests.WorkItemExecution #115531
1043014 dotnet/runtime WasmTestOnChrome-ST-System.Runtime.Tests.WorkItemExecution #114929
1043004 dotnet/runtime WasmTestOnChrome-ST-System.Runtime.Tests.WorkItemExecution #115583
1042941 dotnet/runtime WasmTestOnChrome-ST-System.Runtime.Tests.WorkItemExecution
1042975 dotnet/runtime WasmTestOnChrome-ST-System.Runtime.Intrinsics.Tests.WorkItemExecution #115574
1042908 dotnet/runtime normal-Microsoft.Extensions.Options.SourceGeneration.Unit.Tests.WorkItemExecution
1042879 dotnet/runtime WasmTestOnChrome-ST-System.Runtime.Tests.WorkItemExecution
1042706 dotnet/runtime WasmTestOnChrome-ST-System.Runtime.Tests.WorkItemExecution #115572
1042701 dotnet/runtime WasmTestOnChrome-ST-System.Runtime.Intrinsics.Tests.WorkItemExecution #114427
1042693 dotnet/runtime WasmTestOnChrome-ST-System.Runtime.Tests.WorkItemExecution #115136
1042459 dotnet/runtime WasmTestOnChrome-ST-System.Runtime.Tests.WorkItemExecution #115573
1042628 dotnet/runtime WasmTestOnChrome-ST-System.Runtime.Tests.WorkItemExecution #115616
1042605 dotnet/runtime WasmTestOnChrome-ST-System.Runtime.Tests.WorkItemExecution #115058
1042552 dotnet/runtime WasmTestOnChrome-ST-System.Runtime.Tests.WorkItemExecution #115610
1042549 dotnet/runtime WasmTestOnChrome-ST-System.Runtime.Tests.WorkItemExecution #114148
1042528 dotnet/runtime WasmTestOnChrome-ST-System.Runtime.Tests.WorkItemExecution #115113
1042330 dotnet/runtime WasmTestOnChrome-ST-System.Runtime.Tests.WorkItemExecution #115408
1042424 dotnet/runtime WasmTestOnChrome-ST-System.Runtime.Tests.WorkItemExecution #114901
1042391 dotnet/runtime WasmTestOnChrome-ST-Microsoft.Extensions.Options.SourceGeneration.Unit.Tests.WorkItemExecution
1042376 dotnet/runtime WasmTestOnChrome-ST-System.Runtime.Tests.WorkItemExecution
1042324 dotnet/runtime WasmTestOnChrome-ST-System.Runtime.Tests.WorkItemExecution #113956
1041806 dotnet/runtime WasmTestOnChrome-ST-System.Runtime.Tests.WorkItemExecution #115494
1042273 dotnet/runtime WasmTestOnChrome-ST-System.Runtime.Tests.WorkItemExecution #115590
1042269 dotnet/runtime WasmTestOnChrome-ST-System.Runtime.Tests.WorkItemExecution #114828
1042261 dotnet/runtime WasmTestOnChrome-ST-System.Runtime.Tests.WorkItemExecution #100729
1042237 dotnet/runtime WasmTestOnChrome-ST-System.Runtime.Tests.WorkItemExecution #109138
1042198 dotnet/runtime WasmTestOnChrome-ST-System.Runtime.Tests.WorkItemExecution #115545
1042092 dotnet/runtime WasmTestOnChrome-ST-System.Runtime.Tests.WorkItemExecution #115583
1042076 dotnet/runtime WasmTestOnChrome-ST-System.Runtime.Tests.WorkItemExecution #115590
1042062 dotnet/runtime WasmTestOnChrome-ST-System.Runtime.Tests.WorkItemExecution #115569
1041735 dotnet/runtime WasmTestOnChrome-ST-System.Private.Xml.Tests.WorkItemExecution
1041632 dotnet/runtime WasmTestOnChrome-ST-System.Runtime.Tests.WorkItemExecution #115563
1041752 dotnet/runtime WasmTestOnChrome-ST-System.Runtime.Tests.WorkItemExecution #115531
1041705 dotnet/runtime WasmTestOnChrome-ST-System.Runtime.Tests.WorkItemExecution
1041655 dotnet/runtime WasmTestOnChrome-ST-System.Runtime.Tests.WorkItemExecution #115310
1041639 dotnet/runtime WasmTestOnChrome-ST-System.Runtime.Tests.WorkItemExecution #115574
1041645 dotnet/runtime WasmTestOnChrome-ST-System.Runtime.Tests.WorkItemExecution #115575
1041642 dotnet/runtime WasmTestOnChrome-ST-System.Runtime.Tests.WorkItemExecution #115574
1041628 dotnet/runtime WasmTestOnChrome-ST-System.Runtime.Tests.WorkItemExecution #115573
1041599 dotnet/runtime WasmTestOnChrome-ST-System.Runtime.Tests.WorkItemExecution #114827
1041582 dotnet/runtime WasmTestOnChrome-ST-System.Runtime.Tests.WorkItemExecution #115571
1041547 dotnet/runtime WasmTestOnChrome-ST-System.Runtime.Tests.WorkItemExecution #115569
1041560 dotnet/runtime WasmTestOnChrome-ST-System.Runtime.Tests.WorkItemExecution #115570
1041524 dotnet/runtime WasmTestOnChrome-ST-System.Runtime.Tests.WorkItemExecution #110416
1041490 dotnet/runtime WasmTestOnChrome-ST-System.Runtime.Tests.WorkItemExecution #115408
1041461 dotnet/runtime WasmTestOnChrome-ST-System.Runtime.Tests.WorkItemExecution #115335
1041427 dotnet/runtime WasmTestOnChrome-ST-System.Runtime.Tests.WorkItemExecution #115113
1041375 dotnet/runtime WasmTestOnChrome-ST-System.Runtime.Tests.WorkItemExecution #115548
1041365 dotnet/runtime WasmTestOnChrome-ST-System.Runtime.Tests.WorkItemExecution #115427
1041350 dotnet/runtime WasmTestOnChrome-ST-System.Runtime.Tests.WorkItemExecution #114148
1041309 dotnet/runtime WasmTestOnChrome-ST-System.Runtime.Tests.WorkItemExecution #115513
1041235 dotnet/runtime WasmTestOnChrome-ST-Microsoft.Extensions.Options.SourceGeneration.Unit.Tests.WorkItemExecution
1041247 dotnet/runtime WasmTestOnChrome-ST-System.Runtime.Tests.WorkItemExecution #115427
1041224 dotnet/runtime WasmTestOnChrome-ST-System.Runtime.Tests.WorkItemExecution #115408
1041207 dotnet/runtime WasmTestOnChrome-ST-System.Runtime.Tests.WorkItemExecution
1041108 dotnet/runtime WasmTestOnChrome-ST-System.Runtime.Tests.WorkItemExecution #115548
1041102 dotnet/runtime WasmTestOnChrome-ST-System.Runtime.Tests.WorkItemExecution #114828
1039158 dotnet/runtime WasmTestOnChrome-ST-System.Runtime.Tests.WorkItemExecution #115464
1040895 dotnet/runtime WasmTestOnChrome-ST-System.Runtime.Tests.WorkItemExecution #115538
1040486 dotnet/runtime WasmTestOnChrome-ST-System.Runtime.Tests.WorkItemExecution #115517
1040855 dotnet/runtime WasmTestOnChrome-ST-System.Runtime.Tests.WorkItemExecution #115525
1040810 dotnet/runtime WasmTestOnChrome-ST-System.Runtime.Intrinsics.Tests.WorkItemExecution #115531
1040789 dotnet/runtime WasmTestOnChrome-ST-System.Runtime.Tests.WorkItemExecution #115530
1040760 dotnet/runtime WasmTestOnChrome-ST-Microsoft.Extensions.Configuration.Json.Tests.WorkItemExecution
1040707 dotnet/runtime WasmTestOnChrome-ST-System.Runtime.Tests.WorkItemExecution
1040520 dotnet/runtime WasmTestOnChrome-ST-System.Runtime.Tests.WorkItemExecution #115516
1040497 dotnet/runtime WasmTestOnChrome-ST-System.Runtime.Tests.WorkItemExecution
1040363 dotnet/runtime WasmTestOnChrome-ST-Microsoft.Extensions.Options.SourceGeneration.Unit.Tests.WorkItemExecution
1040339 dotnet/runtime WasmTestOnChrome-ST-System.Runtime.Tests.WorkItemExecution
1040305 dotnet/runtime WasmTestOnChrome-ST-System.Runtime.Tests.WorkItemExecution #115509
1040240 dotnet/runtime WasmTestOnChrome-ST-System.Runtime.Tests.WorkItemExecution #115315
1040208 dotnet/runtime WasmTestOnChrome-ST-System.Runtime.Tests.WorkItemExecution #115465
1040145 dotnet/runtime WasmTestOnChrome-ST-System.Runtime.Tests.WorkItemExecution #114187
1040111 dotnet/runtime WasmTestOnV8-ST-System.Runtime.Tests.WorkItemExecution #115503
1040126 dotnet/runtime normal-System.Runtime.Tests.WorkItemExecution #115502
1040002 dotnet/runtime normal-Microsoft.Extensions.Options.SourceGeneration.Unit.Tests.WorkItemExecution
1040024 dotnet/runtime WasmTestOnChrome-ST-System.Runtime.Tests.WorkItemExecution #114881
1039991 dotnet/runtime WasmTestOnChrome-ST-System.Runtime.Tests.WorkItemExecution #115494
1039888 dotnet/runtime WasmTestOnChrome-ST-System.Collections.Immutable.Tests.WorkItemExecution
1039922 dotnet/runtime WasmTestOnChrome-ST-System.Runtime.Intrinsics.Tests.WorkItemExecution #115436
1039916 dotnet/runtime WasmTestOnChrome-ST-System.Runtime.Tests.WorkItemExecution #115437
1039893 dotnet/runtime WasmTestOnChrome-ST-System.Runtime.Tests.WorkItemExecution #115489
1039841 dotnet/runtime WasmTestOnChrome-ST-System.Runtime.Tests.WorkItemExecution
1039820 dotnet/runtime WasmTestOnChrome-ST-System.Runtime.Tests.WorkItemExecution #115435
Displaying 100 of 492 results

Summary

24-Hour Hit Count 7-Day Hit Count 1-Month Count
0 22 492
@pavelsavara pavelsavara added arch-wasm WebAssembly architecture blocking-clean-ci Blocking PR or rolling runs of 'runtime' or 'runtime-extra-platforms' area-Codegen-AOT-mono wasm-aot-test WebAssembly AOT Test Known Build Error Use this to report build issues in the .NET Helix tab os-browser Browser variant of arch-wasm labels Jul 24, 2023
@pavelsavara pavelsavara added this to the 8.0.0 milestone Jul 24, 2023
@ghost
Copy link

ghost commented Jul 24, 2023

Tagging subscribers to 'arch-wasm': @lewing
See info in area-owners.md if you want to be subscribed.

Issue Details

Error Blob

{
  "BuildRetry": false,
  "ErrorPattern": "emcc.*SIGKILL",
  "ExcludeConsoleLog": true
}

Reproduction Steps

AOT wasm Microsoft.Extensions.Options.SourceGeneration.Unit.Tests

Log

Author: pavelsavara
Assignees: -
Labels:

arch-wasm, blocking-clean-ci, area-Codegen-AOT-mono, wasm-aot-test, Known Build Error, os-browser
Milestone: 8.0.0

@pavelsavara pavelsavara mentioned this issue Jul 24, 2023
Merged
@lewing
Copy link
Member

lewing commented Jul 24, 2023

Is this it hitting the ulimit of the container?

@lewing lewing self-assigned this Jul 24, 2023
@radical
Copy link
Member

radical commented Jul 24, 2023 

[ 7455.672962] [ 2825] 1000 2825 158937 41558 684032 0 0 python3 
[ 7455.672964] [ 4160] 102 4160 67335 453 180224 0 0 rsyslogd 
[ 7455.672966] [ 4181] 0 4181 455864 16951 634880 0 0 amacoreagent 
[ 7455.672968] [ 4453] 0 4453 298556 3977 249856 0 0 auoms 
[ 7455.672969] [ 4495] 0 4495 215902 6240 204800 0 0 auomscollect 
[ 7455.672971] [ 4750] 102 4750 191707 14176 622592 0 0 mdsd 
[ 7455.672973] [ 4967] 0 4967 334103 3030 253952 0 0 azsecd 
[ 7455.672975] [ 5046] 0 5046 352600 4193 270336 0 0 azsecmond 
[ 7455.672977] [ 32457] 0 32457 317321 3577 258048 0 0 azsecd 
[ 7455.672979] [ 2226] 0 2226 376486 334278 2924544 0 0 clamscan 
[ 7455.672981] [ 22067] 0 22067 180189 824 114688 0 -998 containerd-shim 
[ 7455.672983] [ 22089] 1000 22089 1159 16 57344 0 0 helix_docker_wo 
[ 7455.672985] [ 22156] 1000 22156 1159 17 53248 0 0 execute.sh 
[ 7455.672987] [ 22212] 1000 22212 5047 87 81920 0 0 bash 
[ 7455.672989] [ 22215] 1000 22215 68517493 67031 1163264 0 0 dotnet 
[ 7455.672990] [ 23320] 1000 23320 1159 17 61440 0 0 sh 
[ 7455.672992] [ 23321] 1000 23321 34332 22818 335872 0 0 python3 
[ 7455.672994] [ 23345] 1000 23345 1420573 1386396 11264000 0 0 wasm-opt 
[ 7455.672995] oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=/,mems_allowed=0,global_oom,task_memcg=/docker/a2a4080691301bb74e3b282eab0b6f638e9d6f1227e62967b7fd46654706e554,task=wasm-opt,pid=23345,uid=1000 
[ 7455.673018] Out of memory: Killed process 23345 (wasm-opt) total-vm:5682292kB, anon-rss:5545584kB, file-rss:0kB, shmem-rss:0kB, UID:1000 pgtables:11000kB oom_score_adj:0

@lewing
Copy link
Member

lewing commented Jul 25, 2023

lets disable these tests in aot for now

@lewing lewing mentioned this issue Aug 13, 2023
Open
3 tasks
@lewing
Copy link
Member

lewing commented Aug 13, 2023

closting as a duplicate of dotnet/dnceng#450

@lewing lewing closed this as completed Aug 13, 2023
@ghost ghost locked as resolved and limited conversation to collaborators Sep 12, 2023
@pavelsavara
Copy link
Member Author

I see this again https://helixre107v0xdcypoyl9e7f.blob.core.windows.net/dotnet-runtime-refs-pull-99833-merge-af0760c542d34f95a8/normal-System.Runtime.Tests/1/console.f849dad3.log?helixlogtype=result

@pavelsavara pavelsavara reopened this Apr 2, 2024
@lewing
Copy link
Member

lewing commented Apr 24, 2024

Analysis claims there are recent failures but the links are not updated?

@pavelsavara
Copy link
Member Author

we run most of those problematic cases only in runtime-wasm pipeline. Which needs to be triggered manually.
I updated the filter

@lewing lewing removed the blocking-clean-ci Blocking PR or rolling runs of 'runtime' or 'runtime-extra-platforms' label May 2, 2024
@dotnet dotnet unlocked this conversation May 27, 2024
@akoeplinger akoeplinger modified the milestones: 8.0.0, 9.0.0 May 27, 2024
@lewing lewing modified the milestones: 9.0.0, 10.0.0 Aug 15, 2024
@lewing
Copy link
Member

lewing commented Feb 12, 2025

all the recent failures I looked at are in wasm-opt which is known to have pretty degenerate memory behavior when linking with debug symbols. I think we should investigate what we can do here. @radekdoulik lets discuss offline

@lewing lewing assigned radekdoulik and unassigned lewing Feb 12, 2025
@pavelsavara pavelsavara mentioned this issue Mar 21, 2025
Merged
@lewing
Copy link
Member

lewing commented Mar 21, 2025

we hope to see this ramp down now after #113755

@pavelsavara
Copy link
Member Author

pavelsavara commented Mar 21, 2025 •
edited
Loading

Maybe we will need to go down to -O0, I think I saw SIGKILL after #113755 , now we are at -O2

@lewing lewing mentioned this issue Apr 17, 2025
Closed
@lewing
Copy link
Member

lewing commented Apr 18, 2025

all the recent failures I looked at are in wasm-opt which is known to have pretty degenerate memory behavior when linking with debug symbols. I think we should investigate what we can do here. @radekdoulik lets discuss offline

#114805 seems to indicate it does help.

This was referenced May 16, 2025
Merged
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@radekdoulik radekdoulik

Labels
arch-wasm WebAssembly architecture area-Codegen-AOT-mono Known Build Error Use this to report build issues in the .NET Helix tab os-browser Browser variant of arch-wasm wasm-aot-test WebAssembly AOT Test
Projects
None yet
Milestone
10.0.0
Development

No branches or pull requests
5 participants
@radical @lewing @pavelsavara @akoeplinger @radekdoulik