[Bug]: yarn install silently stops if memory is set too low in a CI environment

[macrozone]

Self-service

• I'd be willing to implement a fix

Describe the bug

i migrated a project to yarn 2 with nodeLinker: node-modules

running yarn install on gitlab-ci silently stops:

the log just ends there (followed by the next command)

I had a memory limit of 500Mi, increasing it to 700Mi fixes the problem.

Problem is though that there is no indication of a lack of memory, yarn just silently crashes.

To reproduce

I used node:14 as a base image in gitlab-ci

I set these env vars to limit the memory of the container:

  KUBERNETES_MEMORY_REQUEST: "350Mi"
  KUBERNETES_MEMORY_LIMIT: "500Mi"

running yarn install --immutable in such a container silently stops.

You have to lower the limit probably depending on the size of the project

Environment

System:
    OS: macOS 12.0.1
    CPU: (16) x64 Intel(R) Core(TM) i9-9980HK CPU @ 2.40GHz
  Binaries:
    Node: 14.18.3 - /private/var/folders/h2/6_301c1x0qdbh_3gsyyjyvrr0000gn/T/xfs-d36c185a/node
    Yarn: 3.1.1 - /private/var/folders/h2/6_301c1x0qdbh_3gsyyjyvrr0000gn/T/xfs-d36c185a/yarn
    npm: 6.14.15 - ~/.nvm/versions/node/v14.18.3/bin/npm
  npmPackages:
    jest: ^27.4.5 => 27.4.5

Additional context

No response

[arcanis]

Yarn can't do much, it's your kernel that kills the process with no opportunity to react (if it was Node, you'd have a page fault error with memory addresses etc).

[clhuang]

the issue is not that yarn crashes, it's that it exits with no indication that it fails due to a lack of memory, making it quite difficult to debug. Why would yarn not be able to indicate this?

[clhuang]

if the kernel kills the process, it exits with code 137; yarn instead exits with code 0 so it seems it's not yarn itself that was killed, but one of yarn's child processes, and the overarching yarn process should be able to recognize this and surface the error to the caller.

[arcanis]

Yarn catches error codes from its subprocesses (and it's quite uncommon to have subprocesses during the fetch step), so that's very unlikely to be the case. I don't know exactly why the system would return 0, but my past investigations led me to believe it didn't come from Yarn itself.

[clhuang]

I added the following code to /opt/yarn-v1.22.17/bin/yarn.js:

process.on('exit', code => console.log('HIII not a kernel exit', code))

and it does display the text HIIII not a kernel exit 0, so at the very least, the yarn cli is not being killed by the kernel

[clhuang]

so in the docker container (running with host PID mode), I ran yarn install > /dev/null & sleep 1 && ps aux
which output:

root     25445 10.0  0.1 545852 64328 pts/0    Sl   22:43   0:00 node /opt/yarn-v1.22.17/bin/yarn.js install
root     25464  272  0.7 97015244 241620 pts/0 Rl   22:43   0:02 /usr/local/bin/node /app/.yarn/releases/yarn-3.1.1.cjs install

and then from the host's dmesg I see:

[Thu Mar  3 22:43:49 2022] Memory cgroup out of memory: Killed process 25464 (node) total-vm:97565800kB, anon-rss:994316kB, file-rss:36848kB, shmem-rss:0kB, UID:0 pgtables:7748kB oom_score_adj:0
[Thu Mar  3 22:43:49 2022] oom_reaper: reaped process 25464 (node), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB

so it seems like yarn-3.1.1 is being killed, and the parent yarn-v1.22.17 is the one not recognizing the OOM condition

[clhuang]

so it seems like the issue is here: https://github.com/yarnpkg/yarn/blob/3119382885ea373d3c13d6a846de743eca8c914b/src/cli/index.js#L639
exitCode is null if the child was killed with a SIGKILL, which happens in an OOM condition, so this error just gets swallowed

[ohana54]

Hi, encountered the same issue, could we please reopen this and possibly look at the PR?

[merceyz]

This is both an issue in Yarn Classic and Yarn Berry.
Reproduction for Classic:

docker run -it -w /app -m 128m --rm node:16.18.1 bash
yarn set version berry
yarn init
yarn add typescript

Reproduction for Berry:

docker run -it -w /app -m 128m --rm node:16.18.1 bash
corepack enable
yarn set version berry
yarn init
printf '{"packageManager": "[email protected]"}' > package.json
yarn add typescript

[JeremyLoy]

Hi!

We recently ran into this error when attempting to upgrade from Yarn 1 -> Yarn Berry

First image is a (silently failing) yarn install using berry which quickly consumed 2GB of memory

This second image is a successfully passing yarn install using v1 which consumed less than 1GB of memory. (Sorry that the Y axis is different - we were in the process of right-sizing our CI resources)

Our project has 1,147 total dependencies counted using yarn list --depth=0 | wc -l. So not a lot (by NPM standards at least...)

---

So while the incorrect exit code=0 bug definitely should get fixed, I'm wondering if an alternative angle would be to investigate why memory usage in the new version is so much worse. ~2x memory increase seems to be pretty severe

[trivikr]

Verified that memory usage is high using Resource Usage docker extension while running yarn in v4.0.1 on Node.js v20.9.0.

Steps followed:

docker run -it -w /app -m 128m --rm node:20.9.0 bash
corepack enable
yarn set version berry
yarn init
printf '{"packageManager": "[email protected]"}' > package.json
yarn add typescript

There's a bump in both CPU and memory stats

Screenshot

[sarahsporck]

I also realized this increased memory usage. In my case yarn installs around 600MB of dependencies. During linking memory builds up to around 2.5GB and then falls back down to ~1.3GB after all dependencies are installed. In a local heapsnapshot I could see that there are roughly 60MB on the heap during execution. Therefore, I'm wondering what the 1.2GB (= 2.5GB - 1.3GB) are. Anyone a clue?

[arcanis]

The unzip process is performed in wasm-land (for portability), so the memory grows but never shrinks.

I experimented in #5738 to move this part in JS-land (initially for performances, but I imagine it should decrease the memory usage as well); I'm a little short in time at the moment and had to pause it, but if you or someone else want to play with it that'd be an interesting project!

[mareksuscak]

This needs more attention. It's definitely happening on the latest version of Yarn 4.2.2 for us on CircleCI - on a Medium machine w/ 4GB of memory.

[mifi]

If only there was a way to limit concurrency in yarn. I tried

• networkConcurrency: 1
• taskPoolConcurrency: 1
• taskPoolMode: async
• NODE_OPTIONS='--max-old-space-size=1000'

but yarn always seems to spawn 8 concurrent processes (can be seen in top) and OOMs during the Fetch step. (2GB RAM)

[trivikr]

In aws-samples/aws-sdk-js-notes-app#158, we noticed yarn silently stops the install process on AWS EC2 free-tier t2.micro instances which has 1.0 GiB RAM.