Builtin fetching should be representable by derivations

[roberth]

Is your feature request related to a problem? Please describe.

Built-in fetchers block the evaluator. It's not possible to instantiate a derivation until the hash of the fetched source is computed.

Instead of fetching a source (inputSrc) we could represent built-in fetchers using a derivation, similar to builder = "builtins:fetchurl";.

This has the advantage that fetching the input can be

• delayed
• parallelized
• not be done at all, when dependents are substitutable

Describe the solution you'd like

Represent built-in fetching using derivations.

Fetch these outputs using built-in implementations that are not sandboxed, but trusted by virtue of being in Nix (as we do with fetchTree etc)

When scheduling the derivations, request for the Nix client to perform addToStore instead of scheduling an actual derivation build.

Describe alternatives you've considered

A representation of builtin fetchers that doesn't piggyback on derivations. We'd be reinventing the wheel, and interleaving with drv scheduling would be hard.

Additional context

• fetchTree cleanup #9061
• niv is more efficient than flake = false flake inputs
• Can integrate with Lazy trees #6530 but non-atomic fetching might be harder than in the current situation
  • source filter function needs to keep eval alive (as it currently also does, but it doesn't block)
  • or just fetch it all when filtering needs to happen (as it currently is with FODs)
  • or just let users use ca-derivations to do the filtering

Priorities

Add 👍 to issues you find important.

[Ericson2314]

See also #4095. There are a few other things like this we should take care of.

---

The interaction with lazy trees is very interesting. A FUSE store like Tvix has makes it possible to have demand-driven realization / coterminating derivations in general. You can also imagine an RFC 92-like thing where you produce a directory of symlinks to derivations each of which continues the fetch for that subtree. That works very nicely with fixed output derivations and merkle hashing!

CC @flokli

[roberth]

The FUSE idea is interesting, but I would consider it to be a separate set of features.

[kolloch]

Considering a typical build where I locally invoke nix build (let's call that client-side) but the derivations are build via the local nix daemon or remote builders (let's call them builders):

Would the built-in fetchers still be executed "client-side" or by the builders?

I am asking because one of the special powers of built-in derivations is their access to client-side authentication. If the execution of the fetches would move to the builder, this would often break.

[roberth]

Would the built-in fetchers still be executed "client-side" or by the builders?

Yes, I would like the client to "join" the builders, although this is not the only way to do it.
This exposes the complexity of the hashing trust model. If I get it right off the top of my head, the client can be allowed to build content-addressed paths such as

• content-addressed output derivations
• fixed-output derivations
• builtin fetcher derivations with a verifiable hash (e.g. git tree hash) that's usable as a path hash,
• perhaps builtin fetchers that are trusted, relying on rewriting to make their use safe (ie input addressed derivations need to produce a different hash depending on the result of the trusted builtin fetcher).

Fwiw, it's equivalent to having a remote builder that's not trusted by the store.

one of the special powers of built-in derivations is their access to client-side authentication

Yes, what I'm suggesting will allow the client to perform auth, and without exposing credentials to the daemon. Just like fetching synchronously during eval, but without the synchronicity.

An alternate way to implement this, is to schedule on the client, so that it can choose execute the builtin fetcher derivations on its own, instead of dispatching to the daemon or elsewhere. What's nice about this is that it doesn't require any additions to the daemon's trust logic. Maybe this is the way to go?

[Ericson2314]

content-addressed output derivations are still a bit problematic because while the store object is trustless, the
"derivation, output -> content-address" entries are not.

For pure fetching this is no problem because we are doing fixed output derivations. For impure fetching we'd want to e.g. also return the commit object to e.g. proove the commit hash corresponds to a commit with the given tree hash.

[pwaller]

I came here from #9570. I've prototyped the effect reading the flake json and feeding the information into nixpkgs.fetch*; Fixing this is a big performance improvement for my use case. Being able to avoid fetching (for me, large) input sources means reducing the fetch time for a substituter-hit from minutes to seconds, along with avoiding the additional network use and disk space usage, both of which are very noticable in terms of the user experience. It also fixes the --dry-run output in a big way by making the source fetches visible (rather than having to wait minutes before you can see the dry run output).

In terms of implementation, is there a big reason fetchtree couldn't become builder = "builtin:fetchtree" (similar to <nix/fetchurl>), which then uses the existing fetchtree logic? It seems that would mean that the existing credentials logic would work out of the box. I assume there is a reason relating to lazy trees that this isn't the current approach.

nix/src/libexpr/fetchurl.nix

Line 19 in b1c633c

builder = "builtin:fetchurl";

It seems to me that if fetchTree evaluated to a derivation (with additional metadata) in call-flake it would move fetching to build time:

nix/src/libexpr/flake/call-flake.nix

Line 15 in b1c633c

else fetchTree (node.info or {} // removeAttrs node.locked ["dir"]);

@Ericson2314 also discusses some interaction with cleanSource in #4095 that I don't fully understand, so maybe there are good reasons that what I've suggested is not a simple/complete approach.