hermetic building with e.g. nix

[xwjqv]

Hi, i am trying to build an Observable framework dashboard with nix for an embedded system project.
Nix and other hermetic build tools do not allow downloading in builds, instead all dependencies should have an fixed hash or be build hermetically themselves.

My Problem is that Observable framework always tries to download even if i made the imports explicit and put the dependencies in the node_modules or patched framework's libraries.js to not contain npm:

some hacks i tried to build with nix, probably of no value

the furthest i got was a broken build where duckdb didn't work, most of the time Framework tried to to use the internet but crashed because it couldn't and threw a type error

{
  lib,
  stdenv,
  fetchFromGitHub,
  fetchYarnDeps,
  yarnConfigHook,
  yarnBuildHook,
  yarnInstallHook,
  nodejs,
}:

stdenv.mkDerivation (finalAttrs: {
  pname = "staub-dashboard";
  version = "0.0.0";

  src = ./. ;

  yarnOfflineCache = fetchYarnDeps {
    yarnLock = ./yarn.lock ;# finalAttrs.src + "/yarn.lock";
    #hash = "sha256-AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA=";
    #hash = "sha256-1d+yywqm2td4NyS4++VLeUR2BvJW9QEoy5uWpPMXuvY=";
    #hash = "sha256-Brf4qTq6qkp2k+Y6fmEx6qIVZE+69+MQW10xJVYnc0Y=";
    #hash = "sha256-nh94kCGl42RnofozbdQ9ZeupSmIB3vRVLorMjWyAUDs=";
    #hash = "sha256-y0BL34bCXRyZTtWLLccPMGBU8hgs5XexawM+VV+Phzk=";
    hash = "sha256-1H5lkI/Z9Hw2EuAK4ddzIJRDcetcT0+OMmXt1zf2DMM=";
  };
  yarnKeepDevDeps = true;

  buildPhase = ''
    export HOME=$(mktemp -d)
    LOL=./node_modules/@observablehq/framework/dist/
    mkdir -p ./src/.observablehq/cache/_npm
    cp -r ./node_modules/* ./src/.observablehq/cache/_npm
    mkdir -p ./src/.observablehq/cache/_node
    cp -r ./node_modules/* ./src/.observablehq/cache/_node



    sed -i 's|set.has("npm:@observablehq/duckdb")|false|g' ./node_modules/@observablehq/framework/dist/libraries.js
    sed -i 's/npm://g' ./node_modules/@observablehq/framework/dist/libraries.js
    #sed -i 's/npm://g' ./node_modules/@observablehq/framework/**/*.js

    sed -i 's/npm:react/react/g' ./node_modules/@observablehq/framework/dist/resolvers.js

    sed -i 's/const response = await fetch(href/process.stdout.write(href);const response = await fetch(href/' ./node_modules/@observablehq/framework/dist/npm.js
    #sed -i 's/implicits.add/console.log(Array.from(implicits.keys()))|implicits.add/' ./node_modules/@observablehq/framework/dist/libraries.js
    sed -i 's/globalImports\./(x=>{console.log(globalImports);return globalImports})()./' $LOL/resolvers.js

    sed -i 's/throw new Error(`unhandled implicit download/console.log(`ignoring unhandled implicit download/' $LOL/resolvers.js

    #sed -i 's/implicits.add/console.log(implicits)|implicits.add/g' ./node_modules/@observablehq/framework/dist/libraries.js

    sed -i 's/return implicits/console.log(implicits);return implicits/g' ./node_modules/@observablehq/framework/dist/libraries.js

    yarn --offline run observable --debug build

    cp -r ./node_modules/@observablehq/framework/dist/client/stdlib/* dist/_observablehq/stdlib/
  '';
  NODE_ENV="dev"; 
  DEBUG="*";
  OBSERVABLE_TELEMETRY_DISABLE=true;

  installPhase = ''
  cp -r ./dist/ $out
  '';

  #dontInstall =true;
  nativeBuildInputs = [
    yarnConfigHook
    yarnBuildHook
    yarnInstallHook
    # Needed for executing package.json scripts
    nodejs
  ];

  meta = {
    # ...
  };
})

I was thinking about opening an feature request for locking all dependencies but maybe i missed something

Intended Build process

1. Framework dashboard repo with lock files but no cache
2. Nix fetches Framework itself and it's dependencies, all pinned with npm or yarn lockfiles
3. Nix fetches the dependencies for the dashboard, also pinned and checked with hashes
4. Nix starts isolated build job for the dashboard, only inputs are dependencies and clean project repo
5. build dashboard can be used for other build jobs like my OS image
6. A) change file in project -> jump to 4.
   B) change or update dependency -> jump to 3.

For this to work Framework needs to pin the dependencies

I don't use data loaders because i think they are not compatible with what i want to do

[Fil]

Framework self-hosts all its dependencies, so it should be possible (in theory) to have reproducible builds.

Currently it is trusting cdn.jsdelivr.net to obtain the npm: dependencies as browser-ready ESM, and caches them in ${root}/.observablehq/cache/_npm/. Dependencies that are called directly (without the npm: prefix) are read from node_modules and must be manually installed beforehand.

If you want to skip any download, you certainly have to prime the cache for all these elements (this is also a way to do version control on some of the internal npm: dependencies).

The story with dependencies of dependencies (for instance, the CSS files for KaTeX, or the duckdb-wasm extensions) is slightly more complex, but these files still end up in the same cache. If you provide them to the build, it should not try to download anything.

For this discussion it might help if you could share your intended workflow, the patches you made so far, and the errors you're seeing.