Finding the root cause of an Instance "crash"

[danj3]

Having a consistent problem with a simple component that is written in JS and called a few thousand times. The stack trace output doesn't mean much to me as a guide to where the problem exists. The behavior is that it will run exactly 2473 times and break. I just removed the console output and it runs consistently at 2489 iterations. This seems like a resource exhaustion problem, but what kind and how to change is not clear. Is see the default for EngineConfig consume_fuel is false.

Here is the error message:

Error executing function: error while executing at wasm backtrace:
    0: 0x5866de - <unknown>!<wasm function 9949>
    1: 0x2262d0 - <unknown>!<wasm function 4951>
    2: 0x2541f3 - <unknown>!<wasm function 5216>
    3: 0x254e24 - <unknown>!<wasm function 5219>
    4: 0x255642 - <unknown>!<wasm function 5222>
    5: 0x359eff - <unknown>!<wasm function 7028>
    6: 0x258dbc - <unknown>!<wasm function 5250>
    7: 0x24baa0 - <unknown>!<wasm function 5215>
    8: 0x2459f6 - <unknown>!<wasm function 5213>
    9: 0x2540b9 - <unknown>!<wasm function 5216>
   10: 0x254e24 - <unknown>!<wasm function 5219>
   11: 0x2d5196 - <unknown>!<wasm function 6210>
   12: 0x224b38 - <unknown>!<wasm function 4943>
   13: 0x79b68e - <unknown>!add-origin-country

And the source:

var origin=new Map();

export function addOriginCountry({originCountry}) {
    console.log("a:" + (origin instanceof Map) + ":" + originCountry+"\r");

    try {
        var newCount = (origin.get(originCountry) || 0) + 1;
        origin.set(originCountry, newCount);
        console.log("b:"+originCountry+"="+newCount);
        return  newCount;
    }
    catch (error) {
        console.error("an error! " + error.message);
        return 0;
    }
}

The wit used with componentize:

package example:ex1;

world ex1 {
  record opensky {
    origin-country: string
  }
  export add-origin-country: func(data: opensky) -> u64 ;
}

The try/catch and log messages are all in an attempt to isolate the problem, but have not helped. The console output prints, but I'm not 100% on how the logging propagates through the WASI.

[superchris]

Hmm.. interesting. It seems like this component is stateful in way that I haven't tried to do yet, TBH. OTTOMH it seems like this might be a job for a resource, which is a opaque handle to a stateful thing designed to be shared across the component boundary. This is not yet something we support, thought I would like to add support for it eventually.

In any event, it would be interesting to better understand more specifically what resources (in the generic sense) this component is running out of. I'll see if I can add some better debugging in the rust code and run your example when I get a chance. In the meantime, it would be really interesting to understand what you are trying to do and see if it can be refactored to not depend on the component instance being stateful in this way. The way it is designed a process managing a component will crash and be restarted by its supervisor in the event of function invocation error from wasmtime. This is deliberate, as it seems to get left it in an invalid state by wasmtime in certain cases.

[danj3]

In this example since the global variable isn’t returned or manipulated via the Component interface, I would naively think it would be a global in the wasm and in the instance but not need to be a resource since it is not passed around opaquely. Could the way the Map is updated within the function create memory garbage? I did try fiddling with the memory setting but the outcome was unchanged.

I’m not following your comments on the process being supervised. Are you suggesting that when function call fails like this the elixir caller should recover by starting a new instance and consider it a normal condition? This use of a global variable is an exploration of one possible use case for a Component that needs an internal processing state.
One thing I did notice is if the return value from the function is larger (a whole map instead of here just an integer), it will crash after fewer iterations.

[superchris]

In component_genserver_test.exs there is a test that demonstrates what I'm taking about. If there is an error during the component function execution, I observed that subsequent calls will fail due to the component being left in an invalid state by wasmtime. In this case, letting the component genserver crash and get restarted by its supervisor results in a brand new instance that is usable. TBH, I really hadn't considered how stateful components in his scenario.