Fix for loading wasm files under Node.js and in browser when files we are in another dir (#5368)

With this patch WebAssembly files that are located next to JavaScript files are loaded correctly both in Node.js and in browser, even if the user loads them from a different dir than that one.

We use document.scriptLocation on the web and __dirname in node.js to achieve this. When MODULARIZE is used, we also must handle the case when scriptLocation is no longer present when we call the modularize-generated function later, so we save the location beforehand.

This PR also removes the `Module.*PrefixURL` options, which we numerous and made changes like this hard. Instead, `locateFile` can do all that they can. In ASSERTIONS builds we point people to add, and mention in the docs and changelog. This is also good for code size.

Author: nazar-pc

ChangeLog.markdown

@@ -10,6 +10,10 @@ Not all changes are documented here. In particular, new features, user-oriented
 Current Trunk
 -------------
 
+ - Fix `Module.locateFile` to resolve relative paths to *.wasm, *.mem and other files relatively to JavaScript file rather than current working directory
+ - Add second argument `prefix` to `Module.locateFile` function that contains path to JavaScript file where files are loaded from by default
+ - Remove `Module.*PrefixURL` APIs (use `Module.locateFile` instead)
+
 v1.38.8: 07/06/2018
 -------------------
  - Fix a regression in 1.38.7 with binaryen no longer bundling binaryen.js (which emscripten doesn't need, that's just for handwritten JS users, but emscripten did check for its prescence).

emcc.py

@@ -2602,7 +2602,10 @@ def modularize():
 %(src)s
 
   return %(EXPORT_NAME)s;
-}%(instantiate)s;
+};
+%(EXPORT_NAME)s = %(EXPORT_NAME)s.bind({
+  _currentScript: typeof document !== 'undefined' ? document.currentScript : undefined
+})%(instantiate)s;
 ''' % {
     'EXPORT_NAME': shared.Settings.EXPORT_NAME,
     'src': src,
@@ -2708,11 +2711,7 @@ def generate_html(target, options, js_target, target_basename,
       script.un_src()
       script.inline = ('''
           var memoryInitializer = '%s';
-          if (typeof Module['locateFile'] === 'function') {
-            memoryInitializer = Module['locateFile'](memoryInitializer);
-          } else if (Module['memoryInitializerPrefixURL']) {
-            memoryInitializer = Module['memoryInitializerPrefixURL'] + memoryInitializer;
-          }
+          memoryInitializer = Module['locateFile'] ? Module['locateFile'](memoryInitializer, '') : memoryInitializer;
           Module['memoryInitializerRequestURL'] = memoryInitializer;
           var meminitXHR = Module['memoryInitializerRequest'] = new XMLHttpRequest();
           meminitXHR.open('GET', memoryInitializer, true);

site/source/docs/api_reference/module.rst

@@ -4,7 +4,7 @@
 Module object
 =============
 
-``Module`` is a global JavaScript object with attributes that Emscripten-generated code calls at various points in its execution. 
+``Module`` is a global JavaScript object with attributes that Emscripten-generated code calls at various points in its execution.
 
 Developers can provide an implementation of ``Module`` to control the execution of code. For example, to define how notification messages from Emscripten are displayed, developers implement the :js:attr:`Module.print` attribute.
 
@@ -20,7 +20,7 @@ Developers can provide an implementation of ``Module`` to control the execution
 Creating the Module object
 ==========================
 
-Use emcc's :ref:`pre-js option<emcc-pre-js>` to add JavaScript code that defines (or extends) the ``Module`` object with the behaviour you need. 
+Use emcc's :ref:`pre-js option<emcc-pre-js>` to add JavaScript code that defines (or extends) the ``Module`` object with the behaviour you need.
 
 When generating only JavaScript (as opposed to HTML), no ``Module`` object is created by default, and the behaviour is entirely defined by the developer. For example, creating a ``Module`` object with the following code will cause all notifications from the program to be calls to ``alert()``.
 
@@ -51,15 +51,9 @@ The following ``Module`` attributes affect code execution. Set them to customize
 
 	The commandline arguments. The value of ``arguments`` contains the values returned if compiled code checks ``argc`` and ``argv``.
 
-
-.. js:attribute:: Module.filePackagePrefixURL
-
-	This is the "prefix" URL for a preloaded data file that is hosted separately from its JavaScript and HTML files (it includes the full path up to, but not including, the data file). See :ref:`packaging-files-data-file-location` for more information.
-	
-
 .. js:attribute:: Module.locateFile
 
-	If set, this method will be called when the runtime needs to load a file, such as a ``.wasm`` WebAssembly file, ``.mem`` memory init file, or a file generated by the file packager. The function receives the URL, and should return the actual URL. This lets you host file packages or the ``.mem`` file etc. on a different location than the current directory (which is the default expectation), for example if you want to host them on a CDN. Note that ``locateFile`` is sort of a generalization of ``Module.filePackagePrefixURL``.
+	If set, this method will be called when the runtime needs to load a file, such as a ``.wasm`` WebAssembly file, ``.mem`` memory init file, or a file generated by the file packager. The function receives the relative path to the file as configured in build process and a ``prefix`` (path to JavaScript file), and should return the actual URL. This lets you host file packages or the ``.mem`` file etc. on a different location than the directory of JavaScript file (which is the default expectation), for example if you want to host them on a CDN. NOTE: ``prefix`` might be empty string for memory initializer file is loaded in parallel with JS or in packager, so be careful with those.
 
 .. js:attribute:: Module.logReadFiles
 
@@ -87,7 +81,7 @@ The following ``Module`` attributes affect code execution. Set them to customize
 .. js:attribute:: Module.preInit
 
 	A function (or array of functions) that must be called before global initializers run, but after basic initialization of the JavaScript runtime. This is typically used for :ref:`File System operations <Filesystem-API>`.
-		
+
 .. js:attribute:: Module.preinitializedWebGLContext
 
 	If building with -s GL_PREINITIALIZED_CONTEXT=1 set, you can set ``Module.preinitializedWebGLContext`` to a precreated instance of a WebGL context, which will be used later when initializing WebGL in C/C++ side. Precreating the GL context is useful if doing GL side loading (shader compilation, texture loading etc.) parallel to other page startup actions, and/or for detecting WebGL feature support, such as GL version or compressed texture support up front on a page before or in parallel to loading up any compiled code.
@@ -101,7 +95,7 @@ The following ``Module`` attributes affect code execution. Set them to customize
 .. js:attribute:: Module.print
 
 	Called when something is printed to standard output (stdout)
-	
+
 .. js:attribute:: Module.printErr
 
 	Called when something is printed to standard error (stderr)
@@ -140,7 +134,7 @@ The generated program is able to detect its execution environment by checking th
 
 However, sometimes it may be needed to override the detected environment: a typical use case would be module bundlers (like webpack): they are executed by nodejs but the final output is for browser.
 
-In order to do that, you can dictate your preferred execution environment by setting the ``Module.ENVIRONMENT`` variable to one of those allowed values: 
+In order to do that, you can dictate your preferred execution environment by setting the ``Module.ENVIRONMENT`` variable to one of those allowed values:
 
 ``WEB``
 

site/source/docs/porting/files/packaging_files.rst

@@ -4,10 +4,10 @@
 Packaging Files
 ===============
 
-This topic shows how to package the files that will be used to populate :ref:`Emscripten's virtual file system <file-system-overview>` when the page is loaded. 
+This topic shows how to package the files that will be used to populate :ref:`Emscripten's virtual file system <file-system-overview>` when the page is loaded.
 
 There are two alternatives for how files are packaged: *preloading* and *embedding*. Embedding puts the specified files inside the generated JavaScript, while preloading packages the files separately. Embedding files is much less efficient than preloading and should only be used when packaging small numbers of small files. Preloading also enables the option to separately host the data.
-	
+
 *Emcc* uses the *file packager* to package the files and generate the :ref:`File System API <Filesystem-API>` calls that create and load the file system at run time. While *Emcc* is the recommended tool for packaging, there are cases where it can make sense to run the *file packager* manually.
 
 With ``--use-preload-plugins``, files can be automatically decoded based on
@@ -23,9 +23,9 @@ The command below shows how to package files for preloading:
 .. code-block:: bash
 
     ./emcc file.cpp -o file.html --preload-file asset_dir
-	
+
 The command generates **file.html**, **file.js** and **file.data**. The **.data** file contains all the files in **asset_dir/**, and is loaded by **file.js**.
-	
+
 .. note:: The :ref:`Tutorial <tutorial-files>` demonstrates preloading using the **hello_world_file.cpp** test code.
 
 
@@ -36,7 +36,7 @@ The command for embedding is shown below. In this case *emcc* generates **file.h
     ./emcc file.cpp -o file.html --embed-file asset_dir
 
 
-By default, the files to be packaged should be nested in or below the compile-time command prompt directory. At runtime the same nested file structure is mapped to the virtual file system, with the root corresponding to the command prompt directory. 
+By default, the files to be packaged should be nested in or below the compile-time command prompt directory. At runtime the same nested file structure is mapped to the virtual file system, with the root corresponding to the command prompt directory.
 
 For example, consider a file structure **dir1/dir2/dir3/asset_dir/** where the project is compiled from **dir2**. When we package **asset_dir**, we specify its relative location **dir3/asset_dir/**:
 
@@ -51,35 +51,35 @@ The ``@`` symbol can be used to map packaged files from any location in the loca
 
 .. _packaging-files-file-packager:
 
-Packaging using the file packager tool 
+Packaging using the file packager tool
 ======================================
 
-You can also run the *file packager* manually using the instructions at the top of `file_packager.py <https://github.com/kripken/emscripten/blob/master/tools/file_packager.py>`_. 
+You can also run the *file packager* manually using the instructions at the top of `file_packager.py <https://github.com/kripken/emscripten/blob/master/tools/file_packager.py>`_.
 
 The file packager generates a **.data** file and **.js** file. The **.js** file contains the code to use the data file, and must be loaded *before* loading your main compiled code.
 
 .. note::
 
-	-  Using the *file packager* allows you to run file packaging separately from compiling the code. 
+	-  Using the *file packager* allows you to run file packaging separately from compiling the code.
 	-  You can load multiple datafiles by running the file packager on each and loading the **.js** outputs. See `BananaBread <https://github.com/kripken/BananaBread>`_ for an example of this (`cube2/js/game-setup.js <https://github.com/kripken/BananaBread/blob/master/cube2/js/game-setup.js>`_).
 
-	
+
 .. _packaging-files-data-file-location:
 
 Changing the data file location
 ===============================
 
-By default, the **.data** file containing all the preloaded files is loaded from the same URL as your **.js** file. In some cases it may be useful to have the data file in a different location from the other files — for example if your **.html** and **.js** change a lot you may want to keep the data file on a fast CDN somewhere else. 
+By default, the **.data** file containing all the preloaded files is loaded from the same URL as your **.js** file. In some cases it may be useful to have the data file in a different location from the other files — for example if your **.html** and **.js** change a lot you may want to keep the data file on a fast CDN somewhere else.
 
-This model is supported by changing the :js:attr:`Module.filePackagePrefixURL` to be the URL where the data file is stored (this is a prefix, so should include the full path before the data's file name). The attribute must be specified in a ``<script>`` element before the one that loads the data file.
+This model is supported by specifying :js:attr:`Module.locateFile` function to return URL where the data file is stored. The function must be specified in a ``<script>`` element before the one that loads the data file.
 
 
 .. _packaging-files-packaged-file-location:
 
 Modifying file locations in the virtual file system
 ===================================================
 
-The default approach for packaging is to directly map the nested file structure at compile time — relative to the compile-time command prompt directory — to the root of the virtual file system. The ``@`` symbol can be used in a path at build time to *explicitly* specify where the resource will be located in the virtual file system at runtime. 
+The default approach for packaging is to directly map the nested file structure at compile time — relative to the compile-time command prompt directory — to the root of the virtual file system. The ``@`` symbol can be used in a path at build time to *explicitly* specify where the resource will be located in the virtual file system at runtime.
 
 .. note:: The ``@`` symbol is needed because sometimes it is useful to package files that are *not* nested below the compile-time directory, and for which there is therefore no default mapping to a location in the virtual file system.
 
@@ -102,15 +102,15 @@ Valid Character Set
 ===================
 
 The following characters may be used in filenames: ``A-Z``, ``a-z``, ``0-9``, the space character and any of the characters ``!#$%&'()+,-.;=@[]^_`{}~``. Additionally, the following characters may be used if your host filesystem supports them: ``"*<>?|`` (Windows does not allow using these in filenames). When specifying the character ``@`` on the command line, it must be escaped to the form ``@@`` to avoid triggering the ``src@dst`` mapping notation (see above). The characters ``/``, ``\`` and ``:`` cannot be used.
-	
+
 Monitoring file usage
 =====================
 
-.. important:: Only package the files your app actually needs, in order to reduce download size and improve startup speed. 
+.. important:: Only package the files your app actually needs, in order to reduce download size and improve startup speed.
 
 There is an option to log which files are actually used at runtime. To use it, define the :js:attr:`Module.logReadFiles` object. Each file that is read will be logged to stderr.
 
-An alternative approach is to look at :js:func:`FS.readFiles` in your compiled JavaScript. This is an object with keys for all the files that were read from. You may find it easier to use than logging as it records files rather than potentially multiple file accesses. 
+An alternative approach is to look at :js:func:`FS.readFiles` in your compiled JavaScript. This is an object with keys for all the files that were read from. You may find it easier to use than logging as it records files rather than potentially multiple file accesses.
 
 .. note:: You can also modify the :js:func:`FS.readFiles` object or remove it entirely. This can be useful, say, in order to see which files are read between two points in time in your app.
 
@@ -145,4 +145,4 @@ The following formats are supported:
 Test code
 =========
 
-The `test suite <https://github.com/kripken/emscripten/blob/master/tests/>`_ contains many file packaging examples, and is a good place to search for working code. 
+The `test suite <https://github.com/kripken/emscripten/blob/master/tests/>`_ contains many file packaging examples, and is a good place to search for working code.

site/source/docs/porting/pthreads.rst

@@ -46,7 +46,7 @@ The Emscripten implementation for the pthreads API should follow the POSIX stand
 
 - Note that the function emscripten_num_logical_cores() will always return the value of navigator.hardwareConcurrency, i.e. the number of logical cores on the system, even when shared memory is not supported. This means that it is possible for emscripten_num_logical_cores() to return a value greater than 1, while at the same time emscripten_has_threading_support() can return false. The return value of emscripten_has_threading_support() denotes whether the browser has shared memory support available.
 
-Also note that when compiling code that uses pthreads, an additional JavaScript file `pthread-main.js` is generated alongside the output .js file. That file must be deployed with the rest of the generated code files. By default, `pthread-main.js` will be loaded relative to the main HTML page URL. If it is desirable to load the file from a different location e.g. in a CDN environment, then one can define the `Module.locateFile(filename)` function in the main HTML `Module` object to return the URL of the target location of the `pthread-main.js` entry point. If this function is not defined in `Module`, then the relative location specified by `Module.pthreadMainPrefixURL + '/pthread-main.js'` will be used instead. If this is prefix URL is not specified either, then the default location relative to the main HTML file is used.
+Also note that when compiling code that uses pthreads, an additional JavaScript file `pthread-main.js` is generated alongside the output .js file. That file must be deployed with the rest of the generated code files. By default, `pthread-main.js` will be loaded relative to the main HTML page URL. If it is desirable to load the file from a different location e.g. in a CDN environment, then one can define the `Module.locateFile(filename)` function in the main HTML `Module` object to return the URL of the target location of the `pthread-main.js` entry point. If this function is not defined in `Module`, then the default location relative to the main HTML file is used.
 
 Running code and tests
 ======================

src/Fetch.js

@@ -111,10 +111,9 @@ var Fetch = {
 
       var fetchJs = 'fetch-worker.js';
       // Allow HTML module to configure the location where the 'pthread-main.js' file will be loaded from,
-      // either via Module.locateFile() function, or via Module.pthreadMainPrefixURL string. If neither
-      // of these are passed, then the default URL 'pthread-main.js' relative to the main html file is loaded.
-      if (typeof Module['locateFile'] === 'function') fetchJs = Module['locateFile'](fetchJs);
-      else if (Module['pthreadMainPrefixURL']) fetchJs = Module['pthreadMainPrefixURL'] + fetchJs;
+      // via Module.locateFile() function. If not specified, then the default URL 'pthread-main.js' relative
+      // to the main html file is loaded.
+      fetchJs = locateFile(fetchJs);
       Fetch.worker = new Worker(fetchJs);
       Fetch.worker.onmessage = function(e) {
         out('fetch-worker sent a message: ' + e.filename + ':' + e.lineno + ': ' + e.message);
@@ -557,15 +556,15 @@ function emscripten_start_fetch(fetch, successcb, errorcb, progresscb) {
     } else if (fetchAttrNoDownload) {
       __emscripten_fetch_load_cached_data(Fetch.dbInstance, fetch, reportSuccess, reportError);
     } else if (fetchAttrPersistFile) {
-      __emscripten_fetch_load_cached_data(Fetch.dbInstance, fetch, reportSuccess, performCachedXhr);        
+      __emscripten_fetch_load_cached_data(Fetch.dbInstance, fetch, reportSuccess, performCachedXhr);
     } else {
-      __emscripten_fetch_load_cached_data(Fetch.dbInstance, fetch, reportSuccess, performUncachedXhr);        
+      __emscripten_fetch_load_cached_data(Fetch.dbInstance, fetch, reportSuccess, performUncachedXhr);
     }
   } else if (!fetchAttrNoDownload) {
     if (fetchAttrPersistFile) {
       __emscripten_fetch_xhr(fetch, cacheResultAndReportSuccess, reportError, reportProgress);
     } else {
-      __emscripten_fetch_xhr(fetch, reportSuccess, reportError, reportProgress);        
+      __emscripten_fetch_xhr(fetch, reportSuccess, reportError, reportProgress);
     }
   } else {
 #if FETCH_DEBUG

src/library_debugger_toolkit.js

@@ -445,12 +445,7 @@ var CyberDWARFHeapPrinter = function(cdFileLocation) {
   }
 
   function initialize_debugger(cb) {
-    var cdFile;
-    if (typeof Module['locateFile'] === 'function') {
-      cdFileLocation = Module['locateFile'](cdFileLocation);
-    } else if (Module['cdInitializerPrefixURL']) {
-      cdFileLocation = Module['cdInitializerPrefixURL'] + cdFileLocation;
-    }
+    cdFileLocation = locateFile(cdFileLocation);
     if (ENVIRONMENT_IS_NODE || ENVIRONMENT_IS_SHELL) {
       var data = Module['read'](cdFileLocation);
       install_cyberdwarf(data);