Implement reconstruction of double indirection pointers with C Loader or give tooling on Ports (NodeJS, Python)

[viferga]

🚀 Feature

Right now when calling a C function with an out parameter, for example (char**), it works but there is some details we should review.

For example:

void func(char **str_ptr)
{
   printf("%s\n", *str_ptr); // will print "hello"
    *str_ptr = "world"
}

If we call into this function through MetaCall C API we should do something like:

void *str = metacall_value_create_string("hello", 6);
void *ptr = metacall_value_create_ptr(str);

// Now ptr is equivalent to char** where it points to str when doing *ptr

// Here we call the function (with pseudocode)
metacall("func", ptr);

// At this point *ptr points to "world", but in this case, it is pointing to a C string not a String MetaCall Value
// For solving this, we can just do:
char *world_str = metacall_value_to_ptr(ptr);
metacall_value_create_string(world_string, strlen(world_str));

So now the issue is, we should export this APIs into NodeJS and Python so we can use it like this? Can we make it easier?

I was wondering if it is possible to do this automatically from C land, but I think it's really hard or nearly impossible because only with the type signature (char **), we cannot know exactly the memory layout, how do we know that this is a pointer to a single string, or it is an array of strings? Can we infer this and reconstruct it from the type info received from arguments and the type info inferred from C Loader?

I have started implementing this but I fear there will be cases that this cannot be deduced easily, for example, imagine that we pass a pointer to string in the first case, but we return a pointer to an array of strings, this cannot be inferred in any way, and when reconstructing automatically, we can lose information on that phase.

The best option in my opinion is to leave this to the end user, but if this is the case we will need to implement all the functions (metacall_value_create_*) and so into NodeJS and Python.

Another option we can do is something I was thinking about during a long time, which is making metacall register its own API into MetaCall itself, so with this we can make all the APIs available to every language with a single implementation.

A faster way of doing this can be reusing C Loader but something I do not like about this is the requirement that imposes, you must have the C loader for this compiled.. and I do not like that, I prefer solving this with macro metaprogramming, similar to what we have done in the plugins API (see backtrace_plugin or sandbox_plugin).

I am leaving here some reference:

• Commit: 4d21a73
• Base implementation in C Loader (this will be probably discarded):

  core/source/loaders/c_loader/source/c_loader_impl.cpp

  Line 530 in 4d21a73

  // TODO: This may be too tricky to implement because it is impossible to reconstruct the pointer from the type

• Python test:

  core/source/tests/metacall_python_port_pointer_test/source/metacall_python_port_pointer_test.cpp

  Line 136 in 4d21a73

  // TODO: Implement construction of a pointer after the call

• NodeJS test:

  core/source/tests/metacall_node_port_c_test/source/metacall_node_port_c_test.cpp

  Line 60 in 4d21a73

  // TODO: Test passing reference by arguments string

For equivalent example of how other solve this, here is the equivalent with ctypes on Python:

import ctypes

# Load the shared library
lib = ctypes.CDLL('./libsample.so')

# Set up the function signature
lib.get_message.argtypes = [ctypes.POINTER(ctypes.c_char_p)]
lib.get_message.restype = None

# Prepare the output pointer
result = ctypes.c_char_p()
lib.func(ctypes.byref(result))

# Read the string value
print("Returned string:", result.value.decode('utf-8'))

As you can see, at the end takes the opaque pointer, takes the string pointer a creates a python value with it. Similar as I explained in the code before with MetaCall C API.