Support Nix depexts with `opam env` #5982

RyanGibb · 2024-05-30T19:55:43Z

Nix doesn't install packages in the traditional sense -- instead it puts them in a store and makes them available through environment variables. I.e., nix-shell -p gcc will drop us into a shell with gcc in the $PATH.

We can set appropriate environment variables with Opam to make system dependencies (depexts) available with Nix.
Similar to how nix-shell works under the hood, we create a Nix derivation such as

{ pkgs ? import <nixpkgs> {} }:
with pkgs;
let
  packages = [ gmp ];
  inputs = with buildPackages; packages ++ [ pkg-config ];
in
stdenv.mkDerivation {
  name = "opam-nix-env";
  nativeBuildInputs = inputs;

  phases = [ "buildPhase" ];

  buildPhase = ''
vars=("NIX_CC" "NIX_CC_FLAGS" "NIX_CFLAGS_COMPILE" "NIX_CC_WRAPPER_TARGET_HOST_x86_64_unknown_linux_gnu" "NIX_LDFLAGS" "PKG_CONFIG_PATH")
for var in "''${vars[@]}"; do
  escaped="$(echo "''${!var}" | sed -e 's/^$/@/' -e 's/ /\\ /g')"
  echo "$var    =       $escaped        Nix" >> $out
done
echo "PATH      =       $PATH   Nix"
  '';

  preferLocalBuild = true;
}

Which we can build to output a file with environment variables that make depexts available, in Opam's environment variable format. This file is a Nix store root, so it's dependencies won't be garbage collected by Nix until the file is removed.

This approach came from conversations with @dra27 and is distinct from previous approaches in that it supports providing development environment, which imperative installations with Nix don't #5332 (comment); and doesn't require the user to manage the environment outside of Opam, which would lead to a different workflow #5942.

Initial experiments were done using a package to set such environment variables https://github.com/RyanGibb/nix.opam.
However, in order to work with generic depexts (as opposed to just conf packages), to avoid cyclic dependencies (if conf packages depend on nix.opam but nix.opam depends on conf packages), and due to opam package sandboxing (nix derivation fetching would require opam sandboxing to be disabled or the nix store to be 'primed'), it's better implemented as a depext mechanism.

This has been tested and successfully creates an environment to provide packages conf-gmp and conf-libseccomp, as well as depexts directly.

src/state/opamSysInteract.ml

RyanGibb · 2024-06-25T19:25:26Z

Rebased on master

RyanGibb · 2024-07-06T11:26:35Z

Looks like I'm running into NixOS/nix#10587 with the Docker image

RyanGibb · 2024-07-08T12:21:21Z

src/state/opamSysInteract.ml

+  phases = [ "buildPhase" ];
+
+  buildPhase = ''
+vars=("NIX_CC" "NIX_CC_FLAGS" "NIX_CFLAGS_COMPILE" "NIX_CC_WRAPPER_TARGET_HOST_x86_64_unknown_linux_gnu" "NIX_LDFLAGS" "PKG_CONFIG_PATH")


We might want to make these environment variables configurable in order to support new dependencies from Nix, which might require additional environment variables, without cutting a new release of Opam.

I think capnproto might use a flag that isn't here.

Can we pattern match that NIX_CC_WRAPPER_TARGET_HOST_ - the x86_64_unknown_linux_gnu presumably doesn't want to stay hard-coded in here?

RyanGibb · 2024-07-17T11:36:44Z

Rebased on master -- and could do with another depext CI run if it needs re-approval!

RyanGibb · 2024-07-18T16:21:23Z

Okay! The CI is failing only due to ocaml/opam-repository#26261 (I've got it working locally with my branch of opam-repository)

master_changes.md

src/client/opamSolution.ml

src/state/opamSysInteract.ml

src/state/opamEnv.ml

src/client/opamSolution.ml

kit-ty-kate · 2024-07-18T16:49:49Z

src/client/opamSolution.ml

          OpamSysPkg.(Set.union nf sys.s_not_found)
-        | None -> avail, nf)
-      packages (OpamSysPkg.Set.empty, OpamSysPkg.Set.empty)
+        | None -> avail, req, nf)


i think it might be worth using a dedicated record instead of a tuple at this point. Otherwise it could become confusing very quickly with all those similar types around

We have opamSysPkg.status?

Though, we're also passing around the couple of available and required a lot, which perhaps warrants a separate record.

RyanGibb · 2024-07-23T09:12:39Z

src/state/opamSysInteract.ml

+|} in
+    OpamFilename.write drvFile contents;
+    let envFile = OpamFilename.create dir (OpamFilename.basename (OpamFilename.raw "nix.env")) |> OpamFilename.to_string in
+    [`AsUser "nix-build", [ OpamFilename.to_string drvFile; "--out-link"; envFile ] ], None)


It would be good if we could trigger a re-build of any packages that depend on depexts if the drv path changes, as if they link against a previous derivation it could be GC'd. An alternative might be to keep the old nix.env files around as Nix GC roots.

RyanGibb · 2024-11-19T19:22:51Z

Anyone wanting to play around with this on a NixOS system can try:

$ nix shell github:RyanGibb/nixos/#legacyPackages.x86_64-linux.nixpkgs.opam
$ opam --version
2.3.0

Or add it to your own overlay with:

{
  nixpkgs.overlays = [
    (final: prev: {
      opam = prev.opam.overrideAttrs (_: {
       src = final.fetchurl {
          url = "http://ryan.freumh.org/software/opam-full-2.3.0-nixos-depexts.tar.gz";
          sha256 = "sha256-mRxxZtWFgQ8v1szVq5g5+qVqa+OffoG1aHzGUiMMvT0=";
        };
        version = "2.3.0";
      });
    })
  ];
}

This URL is just the https://github.com/RyanGibb/opam/tree/nixos-depexts-2.3.0 branch with vendored dependencies since I couldn't get them working in Nix,

rjbou · 2025-01-23T18:21:04Z

src/state/opamSysInteract.ml

+with pkgs;
+stdenv.mkDerivation {
+  name = "opam-nix-env";
+  nativeBuildInputs = with buildPackages; [ |} ^ packages ^ {| ];


Is it possible to update that file for each installed package, like that there is no need to retrieve and pass the whole list of already installed packages and their system packages ?

I think this is more ergonomic, as it allows system packages to be uninstalled too. It's bridging opam's imperative mode to nix's declarative model.

.github/scripts/depexts/generate-actions.sh

src/state/opamSysInteract.ml

RyanGibb · 2025-02-03T13:47:11Z

David made an observation in today's Opam dev meeting that this PR will not remove depexts from the Nix derivation until the next install. This might effect reproducibility in some circumstances, but this isn't a huge deal as if Nix users want full reproducibility they can use opam2nix.

kit-ty-kate

while tracking down NixOS/nixpkgs#388471 i tried this branch and it worked really well in a nixos docker container. I was able to build opam-publish which requires openssl and gmp without any issues at all, like magic.

.github/scripts/depexts/generate-actions.sh

We support this by including `all_packages` in a switch in a call to `OpamSolution.get_depexts`, including all system packages for the packages in a switch as `required` in a call to `OpamSysInteract.install, and including all of these in the Nix derivation generate by the Nix depext backend.

Co-authored-by: Kate <[email protected]>

also only attempt to read the nix environment file if the os-family = nixos

kit-ty-kate · 2025-03-09T21:44:42Z

src/state/opamSysInteract.ml

+       let dir = OpamPath.Switch.meta st.switch_global.root st.switch in
+       let drvFile =
+         OpamFilename.create dir
+           (OpamFilename.basename (OpamFilename.raw "env.nix"))


This file seems to be there only as a sort of temporary file for nix-build. I also find the naming convention a bit confusing when taken together with nix.env.
env.nix and nix.env are way too close in terms of name and it's easy to confuse between them.

Wouldn't some sort of nameless temporary file be more indicated if a file has to be used?
Couldn't we use a constant string together with environment variables for packages (and possibly vars) instead?

I have found the env.nix file useful sometimes in debugging what is actually in the nix environment, but I agree the naming is a bit confusing

.github/scripts/depexts/generate-actions.sh

src/state/opamSysInteract.ml

RyanGibb · 2025-03-10T07:49:55Z

Thanks for the comments and commits @kit-ty-kate ! I'm glad it was useful :-) Apologies I haven't had a lot of time to work on this PR recently

kit-ty-kate reviewed Jun 10, 2024

View reviewed changes

src/state/opamSysInteract.ml Outdated Show resolved Hide resolved

RyanGibb force-pushed the nixos-depexts branch from c77db3c to 9e2265e Compare June 13, 2024 08:39

RyanGibb force-pushed the nixos-depexts branch from 9e2265e to 92d90bc Compare June 25, 2024 19:24

RyanGibb force-pushed the nixos-depexts branch 3 times, most recently from 9e67415 to f348b9f Compare June 28, 2024 21:29

RyanGibb force-pushed the nixos-depexts branch from 51dd54b to ee2c280 Compare July 7, 2024 10:12

RyanGibb commented Jul 8, 2024

View reviewed changes

rjbou self-requested a review July 9, 2024 08:59

dra27 mentioned this pull request Jul 15, 2024

External Dependency Not Supported In Flatpak i.e. org.freedesktop.platform #6093

Closed

RyanGibb force-pushed the nixos-depexts branch 5 times, most recently from 1a47e83 to 9892788 Compare July 17, 2024 11:36

RyanGibb force-pushed the nixos-depexts branch 2 times, most recently from ff88713 to 2184b56 Compare July 18, 2024 15:49

kit-ty-kate reviewed Jul 18, 2024

View reviewed changes

RyanGibb commented Jul 23, 2024

View reviewed changes

kit-ty-kate added the AREA: DEPEXTS label Sep 25, 2024

kit-ty-kate added this to the 2.4.0~alpha1 milestone Sep 25, 2024

rjbou reviewed Jan 23, 2025

View reviewed changes

kit-ty-kate requested a review from rjbou February 11, 2025 16:58

kit-ty-kate reviewed Mar 9, 2025

View reviewed changes

.github/scripts/depexts/generate-actions.sh Outdated Show resolved Hide resolved

RyanGibb and others added 14 commits March 9, 2025 18:28

Provide Nix depexts with shell environment variables

032a2e5

Add 'required' to OpamSysInteract.packages_status

5ee9a56

Add Nix depext support to changelog

1aeff86

Adapt the depext docker CI tests for the nix backend

93d964f

Apply suggestions from code review

caf2f44

Co-authored-by: Kate <[email protected]>

remove let opens

ac80c50

refactor nix environment updates into seperate function

a7a7244

also only attempt to read the nix environment file if the os-family = nixos

reformat (80 columns rule + rm tabs)

59ba452

fixup .github/scripts/depexts/generate-actions.sh

6cff046

fixup

4c23497

Make .github/scripts/depexts/generate-actions.sh pass shellcheck

2cca154

fixup: switch --> st + OpamPath.Switch.nix_env

241d29c

fixup: get rid of Option.get

cecc358

kit-ty-kate force-pushed the nixos-depexts branch from 8222184 to cecc358 Compare March 9, 2025 21:35

kit-ty-kate reviewed Mar 9, 2025

View reviewed changes

.github/scripts/depexts/generate-actions.sh Outdated Show resolved Hide resolved

Update .github/scripts/depexts/generate-actions.sh

63b8c18

kit-ty-kate reviewed Mar 9, 2025

View reviewed changes

.github/scripts/depexts/generate-actions.sh Outdated Show resolved Hide resolved

Update .github/scripts/depexts/generate-actions.sh

fb49e5a

kit-ty-kate reviewed Mar 9, 2025

View reviewed changes

src/state/opamSysInteract.ml Outdated Show resolved Hide resolved

fixup add support for opam update --depexts on nix

e2ea120

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Support Nix depexts with `opam env` #5982

Support Nix depexts with `opam env` #5982

RyanGibb commented May 30, 2024 •

edited

Loading

RyanGibb commented Jun 25, 2024

RyanGibb commented Jul 6, 2024

RyanGibb Jul 8, 2024

RyanGibb Feb 3, 2025

dra27 Feb 3, 2025

RyanGibb commented Jul 17, 2024

RyanGibb commented Jul 18, 2024

kit-ty-kate Jul 18, 2024

RyanGibb Jul 18, 2024

RyanGibb Jul 19, 2024

RyanGibb Jul 23, 2024

RyanGibb commented Nov 19, 2024

rjbou Jan 23, 2025

RyanGibb Jan 24, 2025

RyanGibb commented Feb 3, 2025

kit-ty-kate left a comment

kit-ty-kate Mar 9, 2025

RyanGibb Mar 10, 2025

RyanGibb commented Mar 10, 2025

Support Nix depexts with opam env #5982

Are you sure you want to change the base?

Support Nix depexts with opam env #5982

Conversation

RyanGibb commented May 30, 2024 • edited Loading

RyanGibb commented Jun 25, 2024

RyanGibb commented Jul 6, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

RyanGibb commented Jul 17, 2024

RyanGibb commented Jul 18, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

RyanGibb commented Nov 19, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

RyanGibb commented Feb 3, 2025

kit-ty-kate left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

RyanGibb commented Mar 10, 2025

Support Nix depexts with `opam env` #5982

Support Nix depexts with `opam env` #5982

RyanGibb commented May 30, 2024 •

edited

Loading