Restore caching in Nix

[OriolMunoz-da]

See here

If you start working on this, make sure to change the cache version to something like "fixing-nix-cache-MYCOOLNAME-1" first and foremost! If you break the same cache that main uses, it will break all runs. Note that the cache_version appears in several points (at least 6 as of Oct 10) throughout the GHA definitions

The status as of that PR is that we had to disable caching of nix (but not its binary cache!) so that we could finally upgrade the Nix binary version to latest.
Unfortunately, re-enabling the cache breaks things, most often by /nix/store/xyz not found. Unclear if that's just not copied, there's concurrent deletion, or what's up with that.
Copying the /nix/store/ directory instead of mounting also didn't work.

Things that I tried but failed:

• Copying instead of mounting the cache (which was unlikely to have good performance anwyay)
• Excluding the db directory

Nix 2.13.2 was working with caches. 2.13.6 also worked. 2.14 did not. My best guess is it's caused by this change, which is undocumented.

Find the release notes here. Note that, like the above, it might be broken due to an undocumented change... 😢

These are changes that I found suspicious:

Release 2.17 (unlikely to be our problem, but worth mentioning)

Nested dynamic attributes are now merged correctly by the parser. For example:

{
nested = {
foo = 1;
};
nested = {
${"ba" + "r"} = 2;
};
}

This used to silently discard nested.bar, but now behaves as one would expect and evaluates to:

{ nested = { bar = 2; foo = 1; }; }

Note that the feature of merging multiple full declarations of attribute sets like nested in the example is of questionable value. It allows writing expressions that are very hard to read, for instance when there are many lines of code between two declarations of the same attribute. This has been around for a long time and is therefore supported for backwards compatibility, but should not be relied upon.

Instead, consider using the nested attribute path syntax:

{
nested.foo = 1;
nested.${"ba" + "r"} = 2;
}

Release 2.19

When Nix is installed using the binary installer, in supported shells (Bash, Zsh, Fish) XDG_DATA_DIRS is now populated with the path to the /share subdirectory of the current profile. This means that command completion scripts, .desktop files, and similar artifacts installed via nix-env or nix profile (experimental) can be found by any program that follows the XDG Base Directory Specification.

Release 2.29 (this looks bad, but 2.14 was already broken)

No longer copy flakes that are in the nix store #10435 #12877 #13098

Previously, we would duplicate entries like path:/nix/store/* back into the Nix store. This was prominently visible for pinned system flake registry entries in NixOS, e.g., when running nix run nixpkgs#hello.

Release 2.30

Overall breaking, but still doesn't look like our problem: https://nix.dev/manual/nix/2.31/release-notes/rl-2.30.html

Other important knowledge

In PRs
docs saves OSS cache
canton-enterprise saves enterprise cache