Interaction between Rust wasm/foreign modules and encapsulation

[dflemstr]

I hope this is the right place for this discussion; I tried finding a forum that would reach the right target audience and this seemed like the best place.

I've been working with Rust/wasm for quite a while. For that I've been using wasm-bindgen quite heavily (for a not-yet-released project) using a Webpack loader I've authored: https://github.com/dflemstr/rust-native-wasm-loader

One thing I have noticed is that there doesn't seem to be a clear direction regarding how code modules are to be treated between Rust and foreign modules (such as ES6 modules or other wasm modules) on the source level.

Incidentally, it seems to be working similar to C but not consistently. This behavior is quite confusing when interfacing between two languages that both support modules/namespacing (Rust+JS)

Remember, this is the perhaps naive impression of somebody using Rust together with wasm. I'm aware of how the current wasm FFI is closely modeled after the C FFI.

Imagine the following tree (with the module hierarchy like you would expect):

.
└── src
    ├── lib.rs
    ├── reducer
    │   └── editor.js
    └── util
        ├── leftpad.js
        └── leftpad.rs

What should the semantics be around visibility and scope for these files? This is what my intuition would tell me:

• editor.js should import leftpad.rs as ../util/leftpad.rs, or import some myrustlib.wasm module where the symbol is available as util::leftpad::xyz in some fashion.
• leftpad.rs should import editor.js as ../reducer/editor.js.
• (Stretch:) There should be a separate notion of "exported in my Rust crate" and "available to Javascript"
• (Stretch:) It should be possible to mark a symbol as being available only in the util module, so that editor.js would not see such a symbol defined in leftpad.rs, but leftpad.js would be able to.

With that said, here is what current tools are doing:

• Raw rustc symbols that are export "C"'d are exported in a global shared namespace.
• wasm-bindgen will export everything marked with #[wasm_bindgen] also in a global shared namespace (including more advanced things such as structs/classes etc).
• wasm-bindgen will resolve all imports (#[wasm_bindgen(module = "...")]) relative to the location of the final generated module artifact, wherever that may be.
• wasm-bindgen only supports symbols that are both crate-public and exported.
• All imports/exports have no notion of encapsulation.
• (Feel free to suggest other items if we want to make a complete list; I just highlighted some examples)

I find this behavior surprising and feel like one could do either of these things:

• Prohibit imports/exports except on the crate top level (lib.rs/main.rs) and resolve them relative to that location.
• Properly namespace imports/exports such that they behave as I described above, or in some other intuitive fashion.

Does this sound reasonable? What are everybody's thoughts on this?

[mgattozzi]

Hey @dflemstr I would say this is the right place since this is where the working group for wasm and Rust coordinate! I would like to here what @alexcrichton thinks on this considering he's done most of the compiler work regarding the wasm32 target. I'm sure many others here have thoughts on the matter as well!

[Pauan]

@dflemstr You're definitely in the right place. Let me give my thoughts on your post:

editor.js should import leftpad.rs as ../util/leftpad.rs, or import some myrustlib.wasm module where the symbol is available as util::leftpad::xyz in some fashion.

I'm undecided about this. On the one hand importing the .rs file makes sense, but on the other hand Rust compilation units are at the crate level, not the module level.

So I guess it depends on whether we want .js files to be seen as a "part of the crate", or as an external thing which imports the Rust crate. I suppose both use-cases will need to be supported.

leftpad.rs should import editor.js as ../reducer/editor.js.

If we treat .js files as being a part of the crate, then yes absolutely I would expect that.

(Stretch:) There should be a separate notion of "exported in my Rust crate" and "available to Javascript"

I believe Rust already has a notion like that: https://doc.rust-lang.org/book/first-edition/ffi.html#callbacks-from-c-code-to-rust-functions

I imagine it would look like this:

extern "wasm" fn foo() -> i32 {
  5 + 10
}

(Stretch:) It should be possible to mark a symbol as being available only in the util module, so that editor.js would not see such a symbol defined in leftpad.rs, but leftpad.js would be able to.

I'm not so sure about this one. That sounds like it's trying to apply Rust's module system and scoping rules to JS, but JS has its own module system.

---

As for the rest of your post, I generally agree. I think this is less a matter of "the Rust team decided things should be this way" and more a matter of "we haven't decided how things should be yet, so the current status-quo is a bit wonky".

[alexcrichton]

FWIW right now wasm and rustc have no knowledge of import paths really, they're sort of just opaque strings. AFAIK it's bundlers that attach meaning to these import paths to allow files to import one another. In that sense I'm not entirely sure what we'd do differently in rustc/wasm-bindgen/etc?

Specifically on the topic of FFI and "what symbols are exported" or "what is their ABI", that's quite similar to https://github.com/rust-lang-nursery/rust-wasm/issues/29 I think

[Pauan]

@alexcrichton It might be necessary for rustc to carefully generate the relative imports in such a way that they work correctly after running wasm-pack. That will require us to figure out exactly how the custom sections and relative paths work.

[Pauan]

Actually, the situation is much more complicated than I thought. The plan is for rustc to bundle up all the .rs and .js code into a single .wasm file, and then wasm-pack grabs the .js files out of the .wasm file.

But that means if the .js files try to import a .rs file, it doesn't work because the .rs files don't exist!

So we either have to give up on the idea of .js files importing .rs files, or we have to parse the .js and rewrite the imports to point to the .wasm instead.

And then there's issues about name collisions, if multiple Rust crates export the same variable name...

[alexcrichton]

@Pauan oh but my point is that rustc isn't generating imports at all, authors are writing them down in source code and rustc is moving those strings along to the output file.

[Pauan]

@alexcrichton Yes, but there's all sorts of problems if the imports are just passed through unchanged:

1. The author of a crate called mycrate has a src/subdir/foo.rs file and a src/bar.js file.

2. Inside of foo.rs it has #[wasm_import_module = "../bar.js"]

3. When that crate is compiled and run through wasm-pack it gets converted into a mycrate.wasm and bar.js file, but inside of mycrate.wasm it has an import for ../bar.js, which is wrong.

There's a bunch of other nasty stuff that can happen, that's just a small taste.

Doing this right will be hard, since authors will want to import file paths relative to the .rs file, but the compiled .wasm file won't necessarily have the same directory structure as the original .rs file.

[alexcrichton]

Oh sure, but this is also serious hypothetical territory right now. Nothing in the toolchain supports specification of the import module, much less actually resolving it. I think before we debate what should happen we should get something working, see how it works, and go from there.

[CryZe]

Does it makes sense to use a path specific to JS there at all anyway? That means the rust crates would be written with specific JS files in mind, which honestly I don't see working out if we consider normal rust crates like chrono that you have as a dependency. So for something like chrono it makes a lot more sense to just define something like this:

#[wasm_import_module = "chrono"]
extern {
    fn time() -> f64; // Made up signature
}

That way you have a clean namespacing on the rust side that isn't prone to potential conflicts.

However I think it definitely does make sense to be able to specify JS dependencies there too, so it probably makes sense to have a custom section in the wasm that specifies how those map to JS.
WAT-like pseudo-code:

(JSImportMappings ("chrono" "chrono-helper.js"))

And on Rust's side it could look like:

#[js_import_mapping = "chrono-helper.js"]
#[wasm_import_module = "chrono"]
extern {
    fn time() -> f64; // Made up signature
}

[Pauan]

@alexcrichton Absolutely, I'm not trying to stop the prototyping work. But since ES6 modules and npm are stabilized, we can discuss theoretical use-cases, to make sure that we will be able to (eventually) support them.

---

@CryZe That would require that any Rust crate that wants to use JS files would have to publish those JS files as an npm module. That's true even if the Rust crate only wants to include a tiny snippet of JS code.

Also how would the naming go? Would the Rust crate be published to npm as chrono, and the JS snippets published to npm as chrono-js? That seems really weird (and unusual for npm).

And then there's things like wasm-bindgen and stdweb which need to be able to generate JS files. So we will need some way of including JS code in a Rust crate. Maybe it would be inline like the stdweb js! macro, or maybe it would be an external .js file (which is much cleaner), but either way the use case is definitely necessary.

And on Rust's side it could look like [...]

How is that better than specifying a relative path to the .js file? No matter what API we decide on, it will be compiled into wasm custom headers, but your API seems a lot more cumbersome than the relative path API.

---

Here is my proposal: when compiling a crate to wasm32-unknown-*, rustc will add in a custom header to the .wasm file. This header would contain the various metadata we need. This header contains the metadata for all the crates which were compiled (including direct and transitive dependencies).

As such, the header needs to be able to uniquely identify each crate. I assume rustc already has some way of uniquely disambiguating crates, so that can be used. Or perhaps it could use UUIDs. Or it could use absolute filepath mangling. Or it could use an incrementing counter. But I don't really care what method is used, as long as each crate can be uniquely identified.

When a crate uses #[wasm_import_module = "./path/to/foo.js"], rustc would resolve the relative path based upon the .rs file which contains the wasm_import_module. And then it would insert three things into the wasm custom header:

1. The unique identifier for the crate.

2. The absolute filepath to the crate.

3. The relative path from the crate to the .js file which is referenced within wasm_import_module

In addition, when rustc is generating the wasm import for wasm_import_module, it would contain the unique identifier for the crate followed by the relative path from the crate to the .js file.

As an example, let's consider a Rust crate located at /path/to/crate:

// File `src/foo/bar.rs`
#[wasm_import_module = "../qux.js"]
extern {
    fn time() -> f64;
}

#[wasm_import_module = "./corge.js"]
extern {
    fn random() -> f64;
}

// File `src/qux.js`
export function time() {
    return Date.now();
}

// File `src/foo/corge.js`
export function random() {
    return Math.random();
}

And lets suppose that rustc gives the above crate the unique identifier some_unique_identifier

The .wasm custom header would contain this information:

[
    {
        "id": "some_unique_identifier",
        "path": "/path/to/crate",
        "imports": ["src/qux.js", "src/foo/corge.js"]
    }
]

In this example there's only one crate, but keep in mind that it would contain the information for all the crates (including all dependencies).

For the sake of clarity I described the data format using JSON, but I don't really care about the format: it could be binary or anything else.

In addition, when generating the .wasm it would contain the following imports:

(import "some_unique_identifier/src/qux.js" "time" (func time (result f64)))
(import "some_unique_identifier/src/foo/corge.js" "random" (func random (result f64)))

Now that the .wasm file contains the above information, it is now possible for wasm-pack to correctly handle everything. It can parse the .wasm file, extract the information out of the custom header, copy the .js files into the current folder, and then fix the wasm imports to point to the correct filepaths.

Of course wasm-pack will need to handle filename conflicts, and also relative imports in the .js files, but that's an issue for wasm-pack to solve, not rustc. The important thing is that rustc gives wasm-pack enough information so that it's possible for wasm-pack to solve those problems.

Everything I described above doesn't require rustc to do anything complicated: it doesn't need to parse .js files or anything like that.

rustc already knows the absolute filepath to the crate, and it already knows the relative path to the .rs file, so all it needs to do is resolve the wasm_import_module path relative to the .rs file. It's just simple filepath manipulation.

In addition, the above custom header format is generic, so it can work with other languages like C++. In other words, it's not tied specifically to Rust, it simply contains some filepath information, nothing else. So it can become a de-facto wasm standard (similar to the work being done on embedding package.json information inside the wasm custom headers).

And because it simply contains filepaths, it doesn't need to be limited to .js: the same header format could be used to import .wasm files, .css files, etc.

So let's say somebody is compiling to wasm32-unknown-unknown and running it in a non-JS host. They would probably be importing .wasm files rather than .js files, but they can still use wasm_import_module for that, because the header format only cares about filepaths, nothing else.

The above proposal doesn't allow for .js files to import .rs files, but at least it allows .rs files to import .js files. I think importing .rs files inside .js will require a lot more design work (and probably a new header format, separate from the above header format).

Also, I'm not sure if it's even a good idea to allow for importing .rs files within .js, so I don't think it should block the above proposal.

[Pauan]

Oh, also, in my above proposal I assumed that wasm_import_module would be importing relative paths (since that's very intuitive, and consistent with ES6 modules).

I still think that's the best idea, however there is an acceptable alternative:

#[wasm_import_module = "/src/qux.js"]

In this case the import is relative to the crate, not relative to the current .rs file.

The custom header format would be exactly the same, the imports within .wasm would be exactly the same, and wasm-pack would be exactly the same, so the only difference is the syntax of wasm_import_module.

However, I don't like this because it's inconsistent with filepaths in ES6 modules. I think it's really nice if wasm_import_module mirrors ES6 imports as much as possible (especially because we're already committed to importing npm modules).

But if other people are convinced that this approach is better, then I'll accept it. The exact syntax for wasm_import_module isn't that important, the important thing is the support for the custom header format (which is the same regardless of how wasm_import_module works).

[Pauan]

Also, I mentioned the need to uniquely identify each crate. Rather than generating a unique identifier, it could instead simply use the index to uniquely identify each crate.

Looking at the /path/to/crate example again, it could generate this custom header:

[
    {
        "path": "/path/to/crate",
        "imports": ["src/qux.js", "src/foo/corge.js"]
    }
]

And generate these imports within the .wasm:

(import "0/src/qux.js" "time" (func time (result f64)))
(import "0/src/foo/corge.js" "random" (func random (result f64)))

Rather than using some_unique_identifier it is instead using the index within the JSON array (which in this case is 0)

Either approach works fine, but this index-based approach might be easier to implement.