(Android) Native modules: compatability issues with `node-gyp-build`

[achou11]

Attempting to use some modules from the Holepunch ecosystem that have native counterparts but due to assumptions it makes about the project structure, a runtime error occurs related to trying to load the native module:

Error: No native build was found for platform=android arch=arm64 runtime=node abi=93 uv=1 armv=8 libc=glibc node=16.17.1
loaded from: /data/data/com.mapeonext/files/nodejs-project

Will use sodium-native as the example since this error is stemming from there, but this issue applies to any module that uses node-gyp-build for loading native bindings.

Noticed that sodium-native uses __dirname to tell node-gyp-build where to start looking for the native binding:

https://github.com/sodium-friends/sodium-native/blob/b4d2fec3262cb75a5d136046f56b5697606fe252/index.js

Unfortunately, in the context of a NodeJS Mobile React Native project, __dirname resolves to /data/data/com.myproject/files/nodejs-project (as seen in error message above)

This error comes from https://github.com/prebuild/node-gyp-build/blob/8419abba399ec01f28cfb02b207b659153052a69/node-gyp-build.js#L60

Their resolution strategy lives in https://github.com/prebuild/node-gyp-build/blob/8419abba399ec01f28cfb02b207b659153052a69/node-gyp-build.js#L62-L74

My understanding of the directory structure that NodeJS mobile creates for my application is generally as follows (other targets+archs omitted for brevity):

.
├── nodejs-assets/
│  └── nodejs-project/ # contains app code, no  native modules should live here
│     ├── index.js
│     ├── loader.js
│     ├── node_modules/
│     └── package.json
├── nodejs-native-assets/ # contains native modules, separated by platform + architecture
│  └── nodejs-native-assets-arm64-v8a/
│    ├── dir.list
│    ├── file.list
│    └── node_modules/ # bindings for each native module should live here
│       └── sodium-native
│         └── build
│            └── Release
│               └── sodium.node

Wondering how NodeJS Mobile tells the application that it should look in the nodejs-native-assets/nodejs-native-assets-arm64-v8a/node_modules/... directory for loading native bindings.

Think there are a couple of potential solutions:

1. Expose some kind of env variable that points to it, which I can use to patch each module using node-gyp-build to use the relevant native assets directory instead of __dirname. For example, sodium-native index file would look like:

const nativeBindingsDir = path.join(env.NATIVE_ASSETS_DIR, 'node_modules', 'sodium-native') 
module.exports = require('node-gyp-build')(nativeBindingsDir)

2. A module that you can swap out with node-gyp-build that knows about the NodeJS mobile directory structure, which you would use with a bundler. This is similar to what @staltz has for Noderify: https://github.com/staltz/bindings-noderify-nodejs-mobile.

My guess is that 2 is probably a better solution because it wouldn't require patching every module. Only downside is that it assumes that you're using a bundler that allows you to swap out modules, which is recommended but not ideal to assume.

---

Environment info:

OS: macOS 14 (Sonoma)
NodeJS Mobile version: 16.17.10
NPM version: 8.19.4

[achou11]

Ah, looks like I missed an important detail about how nmrn works. Seems like it'll copy the relevant native assets directory and place it in the nodejs-project (at runtime?):

nodejs-mobile-react-native/android/src/main/java/com/janeasystems/rn_nodejs_mobile/RNNodeJsMobileModule.java

Lines 361 to 381 in a3059cc

	private boolean copyNativeAssetsFrom() throws IOException {
	// Load the additional asset folder and files lists
	ArrayList<String> nativeDirs = readFileFromAssets(nativeAssetsPath + "/dir.list");
	ArrayList<String> nativeFiles = readFileFromAssets(nativeAssetsPath + "/file.list");
	// Copy additional asset files to project working folder
	if (nativeFiles.size() > 0) {
	Log.v(TAG, "Building folder hierarchy for " + nativeAssetsPath);
	for (String dir : nativeDirs) {
	new File(nodeJsProjectPath + "/" + dir).mkdirs();
	}
	Log.v(TAG, "Copying assets using file list for " + nativeAssetsPath);
	for (String file : nativeFiles) {
	String src = nativeAssetsPath + "/" + file;
	String dest = nodeJsProjectPath + "/" + file;
	copyAsset(src, dest);
	}
	} else {
	Log.v(TAG, "No assets to copy from " + nativeAssetsPath);
	}
	return true;
	}

So given a dir.list that looks like:

node_modules
node_modules/sodium-native
node_modules/sodium-native/build
node_modules/sodium-native/build/Release

and a file.list that looks like:

node_modules/sodium-native/build/Release/sodium.node

it should produce the following path:

/data/data/com.myapp/files/nodejs-project/node_modules/sodium-native/build/Release/sodium.node

Guess node-gyp-build is assuming that it's being passed /data/data/com.myapp/files/nodejs-project/node_modules/sodium-native while in my case, it's being given /data/data/com.myapp/files/nodejs-project/, which isn't enough for it to work with I guess

Can someone confirm that this file copying only happens at runtime? i.e. I wouldn't be able to analyze the apk and see the copied over files/directories within the nodejs-project directory

[achou11]

Also realizing that part of the issue may be how I'm bundling the app. Using Rollup with the esm-shim plugin, which shims __dirname but to the created bundle file (i.e. absolute path to directory containing index.js) so the original directory references for the modules using that aren't preserved.

Not really sure if there's a way to work around that though, so still need to explore my options.

[staltz]

1. Expose some kind of env variable that points to it
2. A module that you can swap out with node-gyp-build that knows about the NodeJS mobile directory structure

Yes, I think (2) would be ideal because anyway both (1) and (2) require changing (or patching) the native addon repo, which is less than ideal, but (2) hides implementation details more than (1) does (in fact, (1) could be just one way of achieving (2)). In other projects using nodejs-mobile, I also discovered the need for such a package. We basically just need node-gyp-build (or bindings or whatever) to accommodate for the existence of nodejs-mobile.

Guess node-gyp-build is assuming that it's being passed /data/data/com.myapp/files/nodejs-project/node_modules/sodium-native while in my case, it's being given /data/data/com.myapp/files/nodejs-project/, which isn't enough for it to work with I guess

Can someone confirm that this file copying only happens at runtime? i.e. I wouldn't be able to analyze the apk and see the copied over files/directories within the nodejs-project directory

Strong yes to both of these paragraphs. In other projects I have used esbuild and/or patch-package to hack the path pointing to the native addon.

Also realizing that part of the issue may be how I'm bundling the app. Using Rollup with the esm-shim plugin

Might be. I don't have experience with Rollup, and I've used esbuild before which neatly handled __dirname and ESM files.

[staltz]

Note: we have node-gyp-build-mobile that was created to solve a very different problem (patching the bin.js that gets run in the terminal during package install) but could be a good home for new runtime logic.