devenv up stuck

[MarceColl]

Describe the bug
When I run devenv up in a brand new arch linux machine it gets stuck and never finishes at the Building shell stage. The same devenv.nix works in a separate arch linux machine, also up-to-date.

To reproduce
Devenv files: https://gist.github.com/MarceColl/bf5f924cb23b16f2efab99ac5775f451

[seikaku@seikaku seikaku]$ uname -a
Linux seikaku 6.6.52-1-lts #1 SMP PREEMPT_DYNAMIC Wed, 18 Sep 2024 19:02:04 +0000 x86_64 GNU/Linux

I also need --max-jobs=4 in this case for some reason.

[seikaku@seikaku seikaku]$ devenv up --max-jobs=4
• Building processes ...
• Using Cachix: devenv
these 3 derivations will be built:
  /nix/store/q37ckd0zi65qfzkzvzs38akxhbijl3ki-devenv-profile.drv
  /nix/store/8l7v5p81gsfb3zlf27r4sadnabr19nw9-process-compose.yaml.drv
  /nix/store/7zd2ysl9i2fgxrk9gk2drs2v159kxh21-devenv-up.drv
building '/nix/store/q37ckd0zi65qfzkzvzs38akxhbijl3ki-devenv-profile.drv'
created 833 symlinks in user environment
building '/nix/store/8l7v5p81gsfb3zlf27r4sadnabr19nw9-process-compose.yaml.drv'
building '/nix/store/7zd2ysl9i2fgxrk9gk2drs2v159kxh21-devenv-up.drv'
✔ Building processes in 6.1s.
• Starting processes ...
• Building shell ...
building '/nix/store/vjzm8xxzjg7x0y6xkfykndyiqlgxywmy-devenv-shell-env.drv'

User seikaku is part of nixbld.

[seikaku@seikaku seikaku]$ groups
wheel nixbld seikaku

Version

devenv 1.3.0 (x86_64-linux)

[sandydoo]

Does devenv shell work? Also, run it with the --verbose flag to see the nix command it gets stuck on.

[MarceColl]

Sorry, I'm super sleep deprived, I wanted to send this part as well, it gets stuck on

devenv readFile: '«github:cachix/devenv/ef61728d91ad5eb91f86cdbcc16070602e7afa16»/src/modules/latest-version'

Both devenv shell and devenv up get stuck in the same place.

[domenkozar]

I also need --max-jobs=4 in this case for some reason.

Does this mean it only reproduces with 4?

[MarceColl]

It always happens, but if I try it without setting --max-jobs it complains that it cannot build before it even reaches that point.

[seikaku@seikaku seikaku]$ devenv up       
• Building processes ...
• Using Cachix: devenv
these 2 derivations will be built:
  /nix/store/g083qjcf324snh6xx08wfj0gsdjj5qn8-process-compose.yaml.drv
  /nix/store/0iy6yfgp4w2k91dajppgsbic1pchir2h-devenv-up.drv
error:
       error:
                           Unable to start any build;
                           either increase '--max-jobs' or enable remote builds.

                           For more information run 'man nix.conf' and search for '/machines'.
✔ Building processes in 5.3s.
Error:   × Failed to run command `/nix/store/kqn8clfzc13gnqbav7l13r9w7xick9lr-nix-2.24-devenv/bin/nix --show-trace --extra-experimental-features nix-command --extra-experimental-features flakes --option warn-dirty false --keep-going --max-jobs 0 --option
  │ eval-cache false --option extra-substituters https://devenv.cachix.org --option extra-trusted-public-keys devenv.cachix.org-1:w1cLUi8dv3hnoSPGAuibQv+f9TZLr6cv/Hm9XgU50cw= build --no-link --print-out-paths .#procfileScript -vv --log-format
  │ internal-json`
  ╰─▶ Nix command failed: exit status: 1

[sandydoo]

The max-jobs issue is a bug on our end. We'll push a fix for it.

In verbose mode, before the Nix output and the line that hangs, there should be a really long Nix command. That's the command it's running. Try running copy-pasting that and running it separately. It might have a hint for us.

[MarceColl]

Sure, the command previous to the failure is

Running command: /nix/store/kqn8clfzc13gnqbav7l13r9w7xick9lr-nix-2.24-devenv/bin/nix --show-trace --extra-experimental-features nix-command --extra-experimental-features flakes --option warn-dirty false --keep-going --max-jobs 4 --option eval-cache false eval --json .#devenv.processes

[domenkozar]

Pushed a fix for max_jobs in dc7ebaf

[domenkozar]

Sure, the command previous to the failure is

Running command: /nix/store/kqn8clfzc13gnqbav7l13r9w7xick9lr-nix-2.24-devenv/bin/nix --show-trace --extra-experimental-features nix-command --extra-experimental-features flakes --option warn-dirty false --keep-going --max-jobs 4 --option eval-cache false eval --json .#devenv.processes

Does it work if you run it manually?

[MarceColl]

Looks like it:

$ /nix/store/kqn8clfzc13gnqbav7l13r9w7xick9lr-nix-2.24-devenv/bin/nix --show-trace --extra-experimental-features nix-command --extra-experimental-features flakes --option warn-dirty false --keep-going --max-jobs 4 --option eval-cache false eval --json .#devenv.processes
{"postgres":{"exec":"exec /nix/store/wc19qmf9xsry7z9nlcpp9mrwgblcypmj-start-postgres/bin/start-postgres","process-compose":{"availability":{"restart":"on_failure"},"readiness_probe":{"exec":{"command":"/nix/store/b2629mw4kzz7av7m1w9y9g3l103wg2ll-postgresql-15.6/bin/pg_isready -d template1"},"failure_threshold":5,"initial_delay_seconds":2,"period_seconds":10,"success_threshold":1,"timeout_seconds":4},"shutdown":{"signal":2}}}}
[seikaku@seikaku seikaku]$ echo $?
0

[sandydoo]

Are there more in there? Is there one with the command develop in it?

[MarceColl]

Nope, no nix call with develop in it

[domenkozar]

@MarceColl What command hangs if you do devenv --verbose shell? Can you run it manually?

[MarceColl]

[seikaku@seikaku seikaku]$ devenv -v shell 2>&1 | grep command
• Running command: /nix/store/kqn8clfzc13gnqbav7l13r9w7xick9lr-nix-2.24-devenv/bin/nix --show-trace --extra-experimental-features nix-command --extra-experimental-features flakes --option warn-dirty false --keep-going --max-jobs 0 --option eval-cache false eval --json .#devenv.cachix
• Running command: /nix/store/kqn8clfzc13gnqbav7l13r9w7xick9lr-nix-2.24-devenv/bin/nix --show-trace --extra-experimental-features nix-command --extra-experimental-features flakes --option warn-dirty false --keep-going --max-jobs 0 --option eval-cache false store ping --json
• Running command: /nix/store/kqn8clfzc13gnqbav7l13r9w7xick9lr-nix-2.24-devenv/bin/nix --show-trace --extra-experimental-features nix-command --extra-experimental-features flakes --option warn-dirty false --keep-going --max-jobs 0 --option eval-cache false --option extra-substituters https://devenv.cachix.org --option extra-trusted-public-keys devenv.cachix.org-1:w1cLUi8dv3hnoSPGAuibQv+f9TZLr6cv/Hm9XgU50cw= print-dev-env --profile /home/seikaku/seikaku/.devenv/gc/shell

[domenkozar]

@MarceColl if you run the last command manually, does it complete?

[domenkozar]

@MarceColl please book a slot at https://calendly.com/domenkozar and we'll dive into this 1:1

[DGollings]

We're seeing the same issue here, on two machines.
Simply updating our NixOS to the latest version which comes with a newer devenv seems to triggers this.

devenv 1.0.5

[domenkozar]

@DGollings this happens using devenv 1.0.5? The latest should be 1.3.1, what channel are you using?

[DGollings]

yes, so turns out that both colleagues still had old devenv versions installed using nix profile. We removed that and use the 'normal' NixPkgs version and it works now. (I'm not exactly sure which version they have now, latest or almost latest)

So that means that something in the latest devenv breaks old installations? Doesn't explain the issue that @MarceColl has though, but FYI

[DrJKL]

I've also got devenv --verbose shell hanging at

devenv readFile: '«github:cachix/devenv/9d3bf4d4c3fef89d2b6b29ebd64047a29ee932ea»/src/modules/latest-version'

devenv version
devenv 1.3.1 (x86_64-linux)

nix --version
nix (Nix) 2.24.10

[domenkozar]

@DrJKL does it resolve if you do devenv update?

[dunderrrrrr]

Got same issue here on a brand new machine.

emil@public ~/projects> mkdir test2
emil@public ~/projects> cd test2/
emil@public ~/p/test2> devenv init
• Creating devenv.nix
• Creating devenv.yaml
• Creating .envrc
• Creating .gitignore
direnv: loading ~/projects/test2/.envrc
direnv: loading https://raw.githubusercontent.com/cachix/devenv/82c0147677e510b247d8b9165c54f73d32dfd899/direnvrc (sha256-7u4iDd1nZpxL4tCzmPG0dQgC5V+/44Ba+tHkPob1v2k=)
direnv: using devenv
direnv: .envrc changed, reloading
warning: creating lock file '/home/emil/projects/test2/devenv.lock'
• Using Cachix: devenv
direnv: ([/nix/store/jw2ax71m8dzgs8xybginlsyw79ndyww1-direnv-2.35.0/bin/direnv export fish]) is taking a while to execute. Use CTRL-C to give up.
building '/nix/store/mlkghrzjf1ccfkgn6ijxrmr0zyrmiic1-devenv-shell-env.drv'
... [ stuck forver ] ...

Running devenv shell --verbose reveals that it's stuck on:

devenv readFile: '«github:cachix/devenv/991abff153b995192bf36655394246fc97ba8627»/src/modules/latest-version'

$ devenv version
devenv 1.3.1 (x86_64-linux)

$ nix --version
nix (Nix) 2.24.11

[dunderrrrrr]

Got same issue here on a brand new machine.

emil@public ~/projects> mkdir test2
emil@public ~/projects> cd test2/
emil@public ~/p/test2> devenv init
• Creating devenv.nix
• Creating devenv.yaml
• Creating .envrc
• Creating .gitignore
direnv: loading ~/projects/test2/.envrc
direnv: loading https://raw.githubusercontent.com/cachix/devenv/82c0147677e510b247d8b9165c54f73d32dfd899/direnvrc (sha256-7u4iDd1nZpxL4tCzmPG0dQgC5V+/44Ba+tHkPob1v2k=)
direnv: using devenv
direnv: .envrc changed, reloading
warning: creating lock file '/home/emil/projects/test2/devenv.lock'
• Using Cachix: devenv
direnv: ([/nix/store/jw2ax71m8dzgs8xybginlsyw79ndyww1-direnv-2.35.0/bin/direnv export fish]) is taking a while to execute. Use CTRL-C to give up.
building '/nix/store/mlkghrzjf1ccfkgn6ijxrmr0zyrmiic1-devenv-shell-env.drv'
... [ stuck forver ] ...

Running devenv shell --verbose reveals that it's stuck on:

devenv readFile: '«github:cachix/devenv/991abff153b995192bf36655394246fc97ba8627»/src/modules/latest-version'

$ devenv version
devenv 1.3.1 (x86_64-linux)

$ nix --version
nix (Nix) 2.24.11

It seems like this was a ram or cpu issue afterall. Increased my vcpus from 1 to 2 on my vps, also increased ram from 2GB to 4GB and now everything runs fine.

[glencoe]

I ran into the same issue today.
I ran devenv update, devenv gc and finally a plain system reboot solved it.

$ devenv version
devenv 1.3.1 (aarch64-darwin)

$ nix --version
nix (Nix) 2.24.11

[domenkozar]

Can you try devenv from main branch? We've done quite a few fixes.

[dsalaza4]

Also hitting this when using the nixpkgs/devenv:latest@sha256:35875c2cf8f0190888ee141b0071188879bfc85e1285ceeb145d0b1ce451f366 container on a 1vcpu 16gb memory AWS machine.

It hung at:

devenv readFile: '«github:cachix/devenv/cf1d4aa532b3d3169435e26c8d45f11199d4c5d3»/src/modules/latest-version'

$ devenv version
devenv 1.3.1 (aarch64-linux)
$ nix --version
nix (Nix) 2.24.10

[dsalaza4]

I can confirm that this is indeed related to either CPU or RAM as @dunderrrrrr said.

After switching to a 2 vcpu 32gb memory machine, everything ran smoothly.

As I already confirmed it got stuck on a 1 vcpu 16gb memory machine and 16gb is a pretty high number for the use case I was testing, my guess is that for some reason devenv is not working properly on 1 vcpu machines.

[dsalaza4]

I can confirm that this is indeed related to either CPU or RAM as @dunderrrrrr said.

After switching to a 2 vcpu 32gb memory machine, everything ran smoothly.

As I already confirmed it got stuck on a 1 vcpu 16gb memory machine and 16gb is a pretty high number for the use case I was testing, my guess is that for some reason devenv is not working properly on 1 vcpu machines.

Here's a way to reproduce it locally using docker:

my-repo $ docker run --cpus=1 -v $PWD:/my-repo -it nixpkgs/devenv

Once inside devenv container:

cd /my-repo/devenv/path
devenv --verbose shell

[dacevedo12]

Is it a problem with nix or devenv itself? I try running the nix commands it says its running (with --verbose) and they work ok, it just gives a giant bash as the output

[sandydoo]

I managed to replicate the issue when only 1 cpu is available (@dunderrrrrr and @dsalaza4, thanks for investigating this). For a local repro, running devenv with taskset --cpu-list 1 is enough to get it to hang. #1714 should fix that particular case.