Skip to content

Use external Node.js installation via PyPI wheels #182

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
wants to merge 1 commit into
base: main
Choose a base branch
from

Conversation

agriyakhetarpal
Copy link
Member

Description

This PR swaps out our Node.js dependency by including the (unofficial) binaries from PyPI: https://github.com/njzjz/nodejs-wheel. This means that pyodide venv's Python interpreter should no longer need to rely on an Node.js installation.

Closes #119

@agriyakhetarpal agriyakhetarpal changed the title Use external Node.js via PyPI wheels Use external Node.js installation via PyPI wheels Apr 8, 2025
@agriyakhetarpal
Copy link
Member Author

A blocker early into the implementation:

We would still need to address how to make pyodide venv work. When we install pyodide-build in a virtual environment, say, in a folder named venv/, it will now install Node.js only into that virtual environment and not globally.

Now, when we'll create a virtual environment using pyodide venv, say, at .venv-pyodide and then activate it, we won't have node on our PATH as it existed only in the other environment. We'll need to install nodejs-wheel inside the virtual environment.

Then, even if we do so, https://github.com/njzjz/nodejs-wheel/blob/4dfd45dbffbe412dc216746eb71ec1fc73d8ed50/nodejs_wheel/executable.py uses subprocesses, which won't work... their implementation is in contrast to my approach, where I use the execv syscall: https://github.com/agriyakhetarpal/hugo-python-distributions/blob/12aea38b785434c89b92873248a76c69b793cf8c/hugo/cli.py#L68

@agriyakhetarpal
Copy link
Member Author

Never mind – the tests passed, and I'm not sure how. :D

@agriyakhetarpal
Copy link
Member Author

...and it doesn't work, as we don't have wasm32 binaries on PyPI or on jsDelivr:

pip install nodejs-wheel
Looking in indexes: https://pypi.org/simple, file:///Users/agriyakhetarpal/Library/Caches/.pyodide-xbuildenv-0.30.1.dev0+ga4b038c.d20250408/0.27.5/xbuildenv/pyodide-root/package_index
WARNING: Location 'file:/Users/agriyakhetarpal/Library/Caches/.pyodide-xbuildenv-0.30.1.dev0+ga4b038c.d20250408/0.27.5/xbuildenv/pyodide-root/package_index/nodejs-wheel/' is ignored: it is neither a file nor a directory.
Collecting nodejs-wheel
  Downloading nodejs_wheel-22.14.0-py3-none-any.whl.metadata (5.6 kB)
WARNING: Location 'file:/Users/agriyakhetarpal/Library/Caches/.pyodide-xbuildenv-0.30.1.dev0+ga4b038c.d20250408/0.27.5/xbuildenv/pyodide-root/package_index/nodejs-wheel-binaries/' is ignored: it is neither a file nor a directory.
INFO: pip is looking at multiple versions of nodejs-wheel to determine which version is compatible with other requirements. This could take a while.
  Downloading nodejs_wheel-22.13.1-py3-none-any.whl.metadata (5.6 kB)
  Downloading nodejs_wheel-22.13.0-py3-none-any.whl.metadata (5.3 kB)
  Downloading nodejs_wheel-22.12.0-py3-none-any.whl.metadata (5.3 kB)
  Downloading nodejs_wheel-22.11.0-py3-none-any.whl.metadata (3.8 kB)
  Downloading nodejs_wheel-20.18.0-py3-none-any.whl.metadata (3.8 kB)
  Downloading nodejs_wheel-20.17.0-py3-none-any.whl.metadata (3.8 kB)
  Downloading nodejs_wheel-20.16.0-py3-none-any.whl.metadata (3.8 kB)
INFO: pip is still looking at multiple versions of nodejs-wheel to determine which version is compatible with other requirements. This could take a while.
  Downloading nodejs_wheel-20.15.1-py3-none-any.whl.metadata (3.8 kB)
  Downloading nodejs_wheel-20.15.0-py3-none-any.whl.metadata (3.8 kB)
  Downloading nodejs_wheel-20.14.0-py3-none-any.whl.metadata (3.8 kB)
  Downloading nodejs_wheel-20.13.1-py3-none-any.whl.metadata (3.8 kB)
  Downloading nodejs_wheel-20.13.0.post1-py3-none-any.whl.metadata (2.6 kB)
INFO: This is taking longer than usual. You might need to provide the dependency resolver with stricter constraints to reduce runtime. See https://pip.pypa.io/warnings/backtracking for guidance. If you want to abort this run, press Ctrl + C.
ERROR: Cannot install nodejs-wheel==20.13.0.post1, nodejs-wheel==20.13.1, nodejs-wheel==20.14.0, nodejs-wheel==20.15.0, nodejs-wheel==20.15.1, nodejs-wheel==20.16.0, nodejs-wheel==20.17.0, nodejs-wheel==20.18.0, nodejs-wheel==22.11.0, nodejs-wheel==22.12.0, nodejs-wheel==22.13.0, nodejs-wheel==22.13.1 and nodejs-wheel==22.14.0 because these package versions have conflicting dependencies.

The conflict is caused by:
    nodejs-wheel 22.14.0 depends on nodejs-wheel-binaries==22.14.0
    nodejs-wheel 22.13.1 depends on nodejs-wheel-binaries==22.13.1
    nodejs-wheel 22.13.0 depends on nodejs-wheel-binaries==22.13.0
    nodejs-wheel 22.12.0 depends on nodejs-wheel-binaries==22.12.0
    nodejs-wheel 22.11.0 depends on nodejs-wheel-binaries==22.11.0
    nodejs-wheel 20.18.0 depends on nodejs-wheel-binaries==20.18.0
    nodejs-wheel 20.17.0 depends on nodejs-wheel-binaries==20.17.0
    nodejs-wheel 20.16.0 depends on nodejs-wheel-binaries==20.16.0
    nodejs-wheel 20.15.1 depends on nodejs-wheel-binaries==20.15.1
    nodejs-wheel 20.15.0 depends on nodejs-wheel-binaries==20.15.0
    nodejs-wheel 20.14.0 depends on nodejs-wheel-binaries==20.14.0
    nodejs-wheel 20.13.1 depends on nodejs-wheel-binaries==20.13.1
    nodejs-wheel 20.13.0.post1 depends on nodejs-wheel-binaries==20.13.0

To fix this you could try to:
1. loosen the range of package versions you've specified
2. remove package versions to allow pip to attempt to solve the dependency conflict

WARNING: Location 'file:/Users/agriyakhetarpal/Library/Caches/.pyodide-xbuildenv-0.30.1.dev0+ga4b038c.d20250408/0.27.5/xbuildenv/pyodide-root/package_index/pip/' is ignored: it is neither a file nor a directory.
ERROR: ResolutionImpossible: for help visit https://pip.pypa.io/en/latest/topics/dependency-resolution/#dealing-with-dependency-conflicts

Here, we don't particularly need a wasm32 wheel for https://pypi.org/search/?q=+nodejs-wheel-binaries, just a Linux or macOS wheel that gets installed into the venv despite the incompatibility (either through pip install --target or similar).

At this point, I think that this is something that isn't worth doing, unless we want to be super hacky. I'd say it's okay to include a bunch of hacks as we've discussed elsewhere, but this one feels too quirky. We could easily ask users to install Node.js externally and globally, whether it is through Homebrew, https://github.com/njzjz/nodejs-wheel, or whichever package manager they use on their system.

What do you think, @hoodmane?

@ryanking13
Copy link
Member

...and it doesn't work, as we don't have wasm32 binaries on PyPI or on jsDelivr:

I don't understand this. Why is it needed to install nodejs inside the pyodide venv? I thought we need to install it in the host environment.


I'm personally against including Node.js in pyodide-build, for the same reason I'm against including emscripten in pyodide-build. If Node.js is supported by a package with enough trusted people and users (like cmake supported by the scikit-build team), I think it's fine, but, njzjz/nodejs-wheel seems a bit unreliable for us to trust and use.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Should we bundle our own copy of Node.js to work with pyodide venv?
2 participants