Modular Rust Arrow libraries in wasm-bindgen

[kylebarron]

Problem statement

The biggest hurdle with WebAssembly in the browser is that multiple Wasm modules can't share the same memory space. This means that having e.g. parquet-wasm and geoarrow-wasm as two separate NPM modules is annoying! You have to use parquet-wasm to load parquet into Arrow in Wasm... but then copy the data to JS, and then copy it into the next wasm module to do more processing with it! This is slow, memory intensive, and not user friendly.

Solution

In https://github.com/domoritz/arrow-wasm, Dominik's goal appeared to be to see if Arrow in rust/wasm would be faster than Arrow in JS. But since working with raw buffers is pretty fast in JS, it's not surprising that Wasm overhead would outweigh any other speedups.

I think the potential of arrow-wasm instead is in being a foundational library for other wasm-bindgen libraries.

So I see various potential libraries:

• parquet-wasm: depends on arrow-wasm directly; used by consumers who only want to parse Parquet and get it into JS. PR is here: Depend on arrow-wasm parquet-wasm#292
• geoparquet-wasm: depends on arrow-wasm and geoarrow-rs; used by consumers who only want to parse GeoParquet to GeoArrow and get it into JS. PR is here: Use arrow-wasm geoparquet-wasm#6
• geoarrow-wasm-slim: depends on arrow-wasm, doesn't include geoparquet io to keep a small bundle size. Used by consumers who somehow otherwise have their data as geoarrow in the browser.
• geoarrow-wasm-full: includes extras like re-exporting geoparquet-wasm. Used by consumers who want to fetch geoparquet to geoarrow but also do some geospatial operations.

Other libraries for other formats might make sense to add in the future. like geoarrow-flatgeobuf, which uses rust to parse flatgeobuf into geoarrow. Etc.

Drawbacks

• It would be nice if it were possible to add wasm-bindgen methods onto existing structs from another crate, but that doesn't appear to be possible 🥲 . This means that any crates other than arrow-wasm may only use structs defined in arrow-wasm in a functional manner.
• There will be a proliferation of feature flags. E.g. geoarrow-wasm might have feature flags for each compression in parquet-wasm?

cc @H-Plus-Time

[H-Plus-Time]

Yep, this is a good way the way to do things, though I'd add a slight proviso to the drawbacks:

• It isn't possible unless using wrapper structs - at least a fixed 8 byte memory cost per level of wrapping, unsure exactly how much of the function call hopping can be optimized away. Probably 'strongly preferred' would be a better way of terming it.
• Provided there isn't some awful quirk in depending crates, it should at least be relatively boilerplate. The difficulty would probably come from complicated conditional flag combinations (maybe once you're at the level of geopolars or js-polars, the impact of, say, individua compressions at the arrow-wasm/parquet-wasm level are too small to bother with flags other than 'all compressions').

I'd probably also do geoarrow-wasm-slim and geoarrow-wasm-full in the one npm package and manage stuff via exports (~40MB + 2x geoarrow-wasm's size - I reckon all the packages that produce {node,bundler,esm,esm2} bundles can be slimmed down by 25-50% with one straightforward and one less straightforward tweak).

[kylebarron]

• It isn't possible unless using wrapper structs

What I was hoping was that I could re-export all the existing struct's methods, and just add one new method. That doesn't really seem possible without manually wrapping the wrapped-struct's methods one by one?

• Provided there isn't some awful quirk in depending crates, it should at least be relatively boilerplate. The difficulty would probably come from complicated conditional flag combinations (maybe once you're at the level of geopolars or js-polars, the impact of, say, individua compressions at the arrow-wasm/parquet-wasm level are too small to bother with flags other than 'all compressions').

Yeah I agree. It should be straightforward, just annoying.

I reckon all the packages that produce {node,bundler,esm,esm2} bundles can be slimmed down by 25-50% with one straightforward and one less straightforward tweak).

If you have packaging recommendations I'm all ears 🙂 . esm2 was a "temporary" hack to get esm working in deck.gl I believe, or something like that. Because the esm export used syntax only available in some specific environment.

[H-Plus-Time]

Note: this would apply to any other network of wasm-bindgen libraries - this is just the most relevant venue I could find.

Alright, so the getrandom 0.3.0 release is... going swimmingly /s (see uuid-rs/uuid#792 for the resultant dependency hell). Basically, where it was formerly just a feature, it now requires a rustflags cfg (which implies checking in .cargo/config.toml, not ideal), which means it needs to be specified in all crates that might be used as an entrypoint from JS.

I raise it (aside from just a heads up for when some random transitive dependency starts panicking after a routine upgrade (it isn't detectable at compile time 🤦 )) as a concrete instance of the (ironic, given what the W stands for) tension between Web-hosted wasm, and the patchwork of server wasm targets.

Clearly there'll be growing demand for wasi support in general, but with the tiny overlap and decidedly partial feature set, it'll be a fair amount of work; what might be worthwhile is to pick a small subset (say, the sync Vec<u8> -> Table), see how much of a can of worms it ends up being. I personally don't have much use for it (and I gather you don't either), I'm guessing a policy of "contributions welcome, here's a sketch of what we think should be possible, this is what can and cannot be reused between the two targets" would be the best compromise.

[kylebarron]

the getrandom mess is annoying, but posting for my future self that I think I got it working with:

# Cargo.toml
[dependencies]
wasm-bindgen = "0.2.84"

[dependencies.getrandom_v03]
package = "getrandom"
version = "0.3"
features = ["wasm_js"]

[dependencies.getrandom_v02]
package = "getrandom"
version = "0.2"
features = ["js"]

# .cargo/config.toml
[target.wasm32-unknown-unknown]
rustflags = ['--cfg', 'getrandom_backend="wasm_js"']