fix(modules.symlinkScript): substitute replace-literal with sd

[Elias-Graf]

The package "replace" was marked as unfree in NixOS/nixpkgs#516986

[Elias-Graf]

@BirdeeHub I'm fairly sure that this also works for binaries, although I would like to test it. Do we have any?
If not, we should probably use something like substituteInPlace instead.
If yes, we should probably add a check for binaries in checks/filesToPatch.nix.

[BirdeeHub]

we do not have any that we are patching in this repo, but a derivation could have a binary in it, they often do, and filesToPatch should be able to replace strings in it.

We should add a test yes. Maybe add a test where you replace something in the main binary of some program? Like, compile a binary with a drv path from its own drv, and then try to search and replace that drv path with one in the final derivation?

[Elias-Graf]

@BirdeeHub Sorry! I'm afraid I need a little pointer here. Let's say I have the following dummy package:

  dummyPackage =
    (pkgs.runCommand "dummy-app"
      {
        nativeBuildInputs = [ pkgs.makeBinaryWrapper ];
      }
      ''
        mkdir -p $out/bin
        mkdir -p $out/share/applications

        # Wrap hello inside a binary wrapper
        makeWrapper ${pkgs.hello}/bin/hello $out/bin/dummy-app \
          --add-flag "--greeting" \
          --add-flag "Hello, $out"

        # Create a desktop file that references the package path
        cat > $out/share/applications/dummy-app.desktop <<EOF
        [Desktop Entry]
        Name=Dummy App
        Exec=$out/bin/dummy-app
        Icon=$out/share/icons/dummy-app.png
        Type=Application
        EOF
      ''
    )
    // {
      meta.mainProgram = "dummy-app";
    };

This "package" references itself to "print its path".

Now making a wrapper with:

  wrappedPackage = self.lib.evalModule (
    { options, wlib, ... }:
    {
      inherit pkgs;

      imports = [ wlib.modules.default ];

      # filesToPatch = options.filesToPatch.default ++ [ "bin/dummy-app" ];
      filesToPatch = [ "bin/dummy-app" ];

      package = dummyPackage;
    }
  );

Given:

    echo "wrapper: ${wrappedPackage.config.wrapper}";
    echo "wrapper package: ${wrappedPackage.config.package}";
    echo "dummy package: ${dummyPackage}";

I get:

filesToPatch-test> wrapper: /nix/store/3q3vca9q05pqyqnw7dwrp74hj9xabrbw-dummy-app
filesToPatch-test> wrapper package: /nix/store/29mi3q15kbwqcd5n42n7h1f9h6rf6i1m-dummy-app
filesToPatch-test> dummy package: /nix/store/29mi3q15kbwqcd5n42n7h1f9h6rf6i1m-dummy-app

The file that gets patched by saying: filesToPatch = [ "bin/dummy-app" ]; is:

/nix/store/3q3vca9q05pqyqnw7dwrp74hj9xabrbw-dummy-app/bin/dummy-app

Which is just the wrapper generated by this library:

cat /nix/store/3q3vca9q05pqyqnw7dwrp74hj9xabrbw-dummy-app/bin/dummy-app
#!/nix/store/4bwbk4an4bx7cb8xwffghvjjyfyl7m2i-bash-interactive-5.3p9/bin/bash

exec -a "$0" /nix/store/29mi3q15kbwqcd5n42n7h1f9h6rf6i1m-dummy-app/bin/dummy-app  "$@"

What I actually would want to patch is the derivation referenced inside this wrapper:

cat /nix/store/29mi3q15kbwqcd5n42n7h1f9h6rf6i1m-dummy-app/bin/dummy-app
...
# ------------------------------------------------------------------------------------
# The C-code for this binary wrapper has been generated using the following command:


makeCWrapper '/nix/store/a58bx0sw2r7fhk4qyg7wvjdd81zw561h-hello-2.12.3/bin/hello' \
    --add-flag '--greeting' \
    --add-flag 'Hello, /nix/store/29mi3q15kbwqcd5n42n7h1f9h6rf6i1m-dummy-app'


# (Use `nix-shell -p makeBinaryWrapper` to get access to makeCWrapper in your shell)
# ------------------------------------------------------------------------------------

Am I doing/understanding something incorrectly? Is the binary we're calling ever inside "our" derivation? Can we even patch that?

[BirdeeHub]

Ah

So, if the binary you want to patch is in a place we already overwrite, then yeah thats not gonna work...

You would want to wrap pkgs.hello the old fashioned way with pkgs.makeBinaryWrapper, and put into --greeting ${placeholder "out"}, which will refer to THAT derivation. And tell it to create that binary somewhere in the drv that isnt going to be overwritten.

Then you would pass that to config.package, and you should be able to use filesToPatch to patch the placeholder out path in the binary you created to point to the new wrapper derivation instead of the location inside the config.package drv

[Elias-Graf]

I'm not sure if I totally understood everything. Given that we're always replacing the main binary, there would not be any simple way to patch that, no?

If the idea was to "move" the main binary to a different place, have our package be in the original place, and call the moved one, I would have to further look into how to do that.

But simply creating a second binary will solve the issue about it getting overwritten if we only care about that.

I've validated that the tests pass with replace-literal, with sd, and I've validated that the tests fail when using substitute because the binary file contains null bytes.

[BirdeeHub]

   wrappedPackage = self.lib.wrapPackage [ { inherit pkgs; }
    ({ wlib, pkgs, ... }: {
      filesToPatch = [ "bin/fileToBePatched" ];
      package = wlib.wrapPackage [ { inherit pkgs; }
        ({pkgs, ... }: {
          package = pkgs.hello;
          wrapperImplementation = "binary";
          wrapperVariants.fileToBePatched.exePath = "bin/hello"; # <- not the main one, the outer wrapper module will thus not wrap it automatically.
          config.flags."--greeting" = "Hello, ${placeholder "out"}"; # <- maybe stick a \0 at the end here for good measure? Just in case it didnt already have null bytes? IDK?
        })
      ];
    })
  ];

^ maybe something like that? (assuming I did nix syntax on github correctly) but, create a binary somewhere that isnt the main one, then use filesToPatch and patch it, and make sure that the path has been updated to the new one, i.e. the path reported by fileToBePatched will be that of the final wrapper derivation and not the original one?

[zevisert]

It may be that replace is freely re-distributable after all: NixOS/nixpkgs#516986 (comment)

[BirdeeHub]

hmmm...

[BirdeeHub]

#532

^ @MakeShiftArtist points out that gnused if used specifically does actually support binary files.

This sounds like a reasonable option to me too. I still think a test would be nice.

The thing about sd is there are a lot more people who do not already have sd installed. Whereas they probably have gnused installed, probably from that same pkgs version.

[BirdeeHub]

Ill update the tests, then they will tell us.

I didnt notice that you had also done so since I last checked your change, but I updated them to use our new testing lib as well at the same time, so we should go with the test I added.

[BirdeeHub]

Sorry for hijacking your PR, I put you as co-author, I figured, I already accidentally redid half of your PR (the tests) so, may as well just finish it XD