[native] Dynamically Linked Library in Presto CPP #24330

soumiiow · 2025-01-07T17:10:21Z

Description

Depends on facebookincubator/velox#11439 in the Velox space
and based off of the following PR: https://github.com/facebookincubator/velox/pull/1005/files

Motivation and Context

Having these changes will enable users to register custom functions dynamically without requiring a fork of Prestissimo.

Impact

This extends Prestissimo functionality to include dynamic loading of functions, types, connectors, etc.

Test Plan

Unit tested. and Manually end to end tested the changes.

Contributor checklist

Please make sure your submission complies with our contributing guide, in particular code style and commit standards.
PR description addresses the issue accurately and concisely. If the change is non-trivial, a GitHub Issue is referenced.
Documented new properties (with its default value), SQL syntax, functions, or other functionality.
If release notes are required, they follow the release notes guidelines.
Adequate tests were added if applicable.
CI passed.

Release Notes

Please follow release notes guidelines and fill in the release notes below.

== RELEASE NOTES ==

General Changes
* ... :pr:`12345`
* ... :pr:`12345`

Hive Connector Changes
* ... :pr:`12345`
* ... :pr:`12345`

If release note is NOT required, use:

== NO RELEASE NOTE ==

linux-foundation-easycla · 2025-01-07T17:10:27Z

The committers listed above are authorized under a signed CLA.

✅ login: soumiiow / name: Soumya Duriseti (9ef0b63)

czentgr

Please also rebase.

czentgr · 2025-02-03T19:32:42Z

presto-native-execution/presto_cpp/main/CMakeLists.txt

@@ -76,7 +76,8 @@ target_link_libraries(
  ${FOLLY_WITH_DEPENDENCIES}
  ${GLOG}
  ${GFLAGS_LIBRARIES}
-  pthread)
+  pthread
+  velox_dynamic_function_loader)


Move this before the velox_encode library.

czentgr · 2025-02-03T19:33:09Z

presto-native-execution/presto_cpp/main/CMakeLists.txt

@@ -89,6 +90,7 @@ set_property(TARGET presto_server_lib PROPERTY JOB_POOL_LINK
                                               presto_link_job_pool)

 add_executable(presto_server PrestoMain.cpp)
+target_link_options(presto_server BEFORE PUBLIC "-Wl,-export-dynamic")


Can we add a comment here why we need these flags?

czentgr · 2025-02-03T19:35:24Z

presto-native-execution/presto_cpp/main/PrestoServer.cpp

+    const fs::path path(systemConfig->pluginDir());
+    PRESTO_STARTUP_LOG(INFO) << path;
+    std::error_code
+        ec; // For using the non-throwing overloads of functions below.


We don;t need this comment here and so can fix up the odd formatting.

czentgr · 2025-02-03T19:35:52Z

presto-native-execution/presto_cpp/main/PrestoServer.cpp

+void PrestoServer::registerDynamicFunctions() {
+  auto systemConfig = SystemConfig::instance();
+  if (!systemConfig->pluginDir().empty()) {
+    // if it is a valid directory, traverse and call dynamic function loader


Please make sure the comments are full sentences beginning with capitalization etc.

czentgr · 2025-02-03T19:37:42Z

presto-native-execution/presto_cpp/main/PrestoServer.cpp

+        auto dirEntryPath = dirEntry.path();
+        if (!fs::is_directory(dirEntry, ec) &&
+            extensions.find(dirEntryPath.extension()) != extensions.end()) {
+          facebook::velox::loadDynamicLibrary(dirEntryPath.c_str());


facebook is not needed here because we are already in the facebook namespace.

steveburnett

Thanks for the doc! A few minor formatting and phrasing suggestions.

steveburnett · 2025-02-04T14:43:55Z

presto-native-execution/presto_cpp/main/dynamic_registry/README.md

@@ -0,0 +1,17 @@
+# Dynamic Loading of Presto Cpp Extensions


Suggested change

# Dynamic Loading of Presto Cpp Extensions

# Dynamic Loading of Presto CPP Extensions

steveburnett · 2025-02-04T14:44:16Z

presto-native-execution/presto_cpp/main/dynamic_registry/README.md

@@ -0,0 +1,17 @@
+# Dynamic Loading of Presto Cpp Extensions
+This library adds the ability to load User Defined Functions (UDFs), connectors, or types without having to fork and build Prestissimo, through the use of shared libraries that a Prestissimo worker can access. These are to be loaded on launch of the Presto server. The Presto server searches for any .so or .dylib files and loads them using this library.


Suggested change

This library adds the ability to load User Defined Functions (UDFs), connectors, or types without having to fork and build Prestissimo, through the use of shared libraries that a Prestissimo worker can access. These are to be loaded on launch of the Presto server. The Presto server searches for any .so or .dylib files and loads them using this library.

This library adds the ability to load User Defined Functions (UDFs), connectors, or types without having to fork and build Prestissimo, through the use of shared libraries that a Prestissimo worker can access. These are loaded on launch of the Presto server. The Presto server searches for any .so or .dylib files and loads them using this library.

steveburnett · 2025-02-04T14:45:42Z

presto-native-execution/presto_cpp/main/dynamic_registry/README.md

+This library adds the ability to load User Defined Functions (UDFs), connectors, or types without having to fork and build Prestissimo, through the use of shared libraries that a Prestissimo worker can access. These are to be loaded on launch of the Presto server. The Presto server searches for any .so or .dylib files and loads them using this library.
+## Getting started
+1. Create a cpp file for your dynamic library
+For dynamically loaded function registration, the format followed is mirrored of that of built-in function registration with some noted differences. Using [MyDynamicFunction.cpp](examples/MyDynamicFunction.cpp) as an example, the function uses the extern "C" keyword to protect against name mangling. A registry() function call is also necessary here.


Suggested change

For dynamically loaded function registration, the format followed is mirrored of that of built-in function registration with some noted differences. Using [MyDynamicFunction.cpp](examples/MyDynamicFunction.cpp) as an example, the function uses the extern "C" keyword to protect against name mangling. A registry() function call is also necessary here.

For dynamically loaded function registration, the format is similar to that of built-in function registration, with some noted differences. Using [MyDynamicFunction.cpp](examples/MyDynamicFunction.cpp) as an example, the function uses the extern "C" keyword to protect against name mangling. A registry() function call is also necessary here.

presto-native-execution/presto_cpp/main/dynamic_registry/README.md

steveburnett · 2025-02-04T14:54:55Z

presto-native-execution/presto_cpp/main/dynamic_registry/README.md

+```
+plugin.dir="User\Test\Path\plugin"
+```
+4. When the worker or the sidecar process starts, it will scan the plugin directory and attempt to dynamically load all shared libraries


Suggested change

4. When the worker or the sidecar process starts, it will scan the plugin directory and attempt to dynamically load all shared libraries

When the worker or the sidecar process starts, it scans the plugin directory and attempts to dynamically load all shared libraries.

I don't think of this as part of the steps to configure, it's what happens when things are run using the config in steps 1-3.

steveburnett

Thanks for the documentation! I really liked your including the setup steps on plugin.rst. Some minor suggestions for formatting and phrasing but looks good overall.

presto-native-execution/presto_cpp/main/dynamic_registry/README.md

presto-docs/src/main/sphinx/presto_cpp/plugin.rst

presto-docs/src/main/sphinx/presto_cpp/plugin/function_plugin.rst

soumiiow · 2025-02-11T22:24:12Z

Thanks @steveburnett please take another look, I've made the changes. I'm not sure about the tone on the intro to UDFs i have on function_plugin.rst, would appreciate another set of eyes there.

steveburnett

Thanks for the revision! Nice work, your unordered list of UDF benefits was great and the format fixes in the README look good.

I made a couple of small suggestions about the intro to UDFs, let me know what you think.

presto-docs/src/main/sphinx/presto_cpp/plugin/function_plugin.rst

steveburnett

I should have noticed this spelling nit earlier! After I found this one, I did a complete review of the doc in this PR and found no other errors so I think this is the last one.

presto-docs/src/main/sphinx/presto_cpp/plugin/function_plugin.rst

aditi-pandit

@soumiiow : Thanks for this code. Would be great to have an e2e test with a SQL statement using the dynamic functions. On the lines of https://github.com/prestodb/presto/blob/master/presto-native-execution/src/test/java/com/facebook/presto/nativeworker/TestPrestoNativeRemoteFunctions.java

aditi-pandit · 2025-02-14T22:53:47Z

presto-native-execution/presto_cpp/main/dynamic_registry/DynamicFunctionRegistrar.h

+    auto systemConfig = SystemConfig::instance();
+    cpp_nameSpace = systemConfig->prestoDefaultNamespacePrefix();
+  }
+  std::string cpp_name(cpp_nameSpace);


Is there an assumption that namespace should end in "." ? We should make this behavior more robust by adding the period separator if its not present.

aditi-pandit · 2025-02-14T22:54:20Z

presto-native-execution/presto_cpp/main/dynamic_registry/DynamicFunctionRegistrar.h

+    const char* nameSpace = "",
+    const std::vector<velox::exec::SignatureVariable>& constraints = {},
+    bool overwrite = true) {
+  std::string cpp_nameSpace(nameSpace);


use camelcase cppNamespace

aditi-pandit · 2025-02-14T22:56:25Z

presto-native-execution/presto_cpp/main/PrestoServer.cpp

@@ -1598,5 +1600,28 @@ protocol::NodeStatus PrestoServer::fetchNodeStatus() {

  return nodeStatus;
 }
+void PrestoServer::registerDynamicFunctions() {
+  auto systemConfig = SystemConfig::instance();


Might be good to have a config of library names to load rather than load all those found in the directory. wdyt ?

@aditi-pandit I'm trying to understand here what would the additional benefit be in doing so.

From the perspective of the user, only the shared library paths will be recognized here so any say .txt files, .cpp files, etc will not be read. Beyond that, is there a benefit to having a config here? I see it as another step for the user to have to maintain and worry about every time they'd want to add/remove a shared library

@soumiiow : My thinking is that having a config of library names could help us if someone accidentally put in a library that had errors and issues. Right now a bad library causes the entire loading to fail, and the platform owner would need to restage the complete directory. If on the other hand they have a white-list of libraries that are good, they can easily add or remove libraries by changing the config.

I agree with @aditi-pandit , better to have a config file/entry to limit the library that can be loaded by prestissimo. Imagine that you load all libraries under a specific folder, what if someone write a library maliciously and put it into this folder. When calling those function that might lead to:

Process crash

Same privilege is shared, the function can do anything they want.

And if scanned by a security tool, this piece of code could report a vulnerability.

BTW, if a customer implement a new UDF themself and put the library to this plugin directory, do they need to restart presto server to take effect?

@PingLiuPing to answer the question, yes they have to restart the coordinator and all workers for this new udf to take effect

Having recursive globbing or regex can remediate the issue and that's a good idea.
But I think your current logic is good enough to filter the dynamic libraries from other file types. And you have defined the plugin.dir in configuration file.

You need to verify each dynamic library is actually the one you want to load. The easist way to achieve this is adding a verify functioin inside the dynamic library. And when loading the library you can check if verify function is existed and then execute it. Inside verify function you can check the pass-in value against a fixed value, for example a UUID or even the UDF name. Though this still can be compromised, a step further towards industry product.

I just jump in to your new feature. I assume the UDF is automatically registerd.
Another approach is provide another layer of control where after the function been loaded into velox, those function cannot be directly registed and called. Instead, some DDL need to be executed to register those function, for example
create function ABC (parameters...) options (library_path ......)
In this way you define the library path explicitly for each function. And only those function been explicitly created by above DDL can be called.

Your work opened up a whole new world. And it deserves to be more careful. Welcome to discuss more alternatives.
Thanks.
@soumiiow @mohsaka

aditi-pandit · 2025-02-14T22:58:31Z

presto-native-execution/presto_cpp/main/dynamic_registry/examples/MyDynamicArrayFunction.cpp

Please add an example of a function which is added to the non-default namespace.

mohsaka · 2025-02-17T19:32:46Z

Changes needed for Mac,

Undefined symbols for architecture arm64:
  "facebook::presto::SystemConfig::instance()", referenced from:
      void facebook::presto::registerPrestoFunction<facebook::velox::common::dynamicRegistry::Dynamic123Function, facebook::velox::Varchar>(char const*, char const*, std::__1::vector<facebook::velox::exec::SignatureVariable, std::__1::allocator<facebook::velox::exec::SignatureVariable>> const&, bool) in MyDynamicVarcharFunction.cpp.o
  "facebook::presto::SystemConfig::prestoDefaultNamespacePrefix() const", referenced from:
      void facebook::presto::registerPrestoFunction<facebook::velox::common::dynamicRegistry::Dynamic123Function, facebook::velox::Varchar>(char const*, char const*, std::__1::vector<facebook::velox::exec::SignatureVariable, std::__1::allocator<facebook::velox::exec::SignatureVariable>> const&, bool) in MyDynamicVarcharFunction.cpp.o
ld: symbol(s) not found for architecture arm64
c++: error: linker command failed with exit code 1 (use -v to see invocation)
ninja: build stopped: subcommand failed.

Add presto_common and remove all other linked libraries as they are included with it.

Remove braces from examples.

Update:
Put target_link_options to avoid symbol errors.

if(APPLE)
  set(COMMON_LIBRARY_LINK_OPTIONS "-Wl,-undefined,dynamic_lookup")
else()
  set(COMMON_LIBRARY_LINK_OPTIONS "-Wl,--exclude-libs,ALL")
endif()

target_link_options(velox_function_my_dynamic PRIVATE ${COMMON_LIBRARY_LINK_OPTIONS})
target_link_options(velox_varchar_function_my_dynamic PRIVATE ${COMMON_LIBRARY_LINK_OPTIONS})
target_link_options(velox_array_function_my_dynamic PRIVATE ${COMMON_LIBRARY_LINK_OPTIONS})

PingLiuPing · 2025-02-19T17:18:39Z

presto-native-execution/presto_cpp/main/PrestoServer.cpp

@@ -1598,5 +1600,28 @@ protocol::NodeStatus PrestoServer::fetchNodeStatus() {

  return nodeStatus;
 }
+void PrestoServer::registerDynamicFunctions() {
+  auto systemConfig = SystemConfig::instance();


I agree with @aditi-pandit , better to have a config file/entry to limit the library that can be loaded by prestissimo. Imagine that you load all libraries under a specific folder, what if someone write a library maliciously and put it into this folder. When calling those function that might lead to:

Process crash

Same privilege is shared, the function can do anything they want.

And if scanned by a security tool, this piece of code could report a vulnerability.

BTW, if a customer implement a new UDF themself and put the library to this plugin directory, do they need to restart presto server to take effect?

PingLiuPing · 2025-02-19T17:18:55Z

presto-docs/src/main/sphinx/presto_cpp/plugin/function_plugin.rst

+      extern "C" {
+         void registry() {
+            facebook::presto::registerPrestoFunction<
+               nameOfStruct,


nit: nameOfStruct to NameOfStruct.

PingLiuPing · 2025-02-19T17:19:09Z

presto-docs/src/main/sphinx/presto_cpp/plugin/function_plugin.rst

+* Once defined, easily reusable and called multiple times just like built in functions.
+* Shorter compile times.
+
+1. To create the UDF, create a new C++ file in the following format:


I think it is better to give a file name here, that will better align wiht the following content. For exmaple TestFunction.cpp is ok.

mohsaka · 2025-02-19T17:58:40Z

@PingLiuPing @aditi-pandit

Ideally we would have a kind of recursive globbing but there doesn't seem to be any library for that. Then we would have a config that allows something like this "a/b/c.dylib a//.dylib etc". But this will need to be implemented with probably filesystem + glob libraries.

Alternatively we can use the regular globbing library and have the user have to enter the directories. So then it would be something like "a/b/* a/c/* etc"

The easiest would be requiring the user to have to provide the absolute path for all of the libraries. Something like "a/b/c.dylib a/b/d.dylib"

An alternative would be that you can provide a path regex. Something like "^/a/b/.*" and a directory. We would then recursively go through the directory and only pick libraries that match a certain regex expression.

Or even simpler, just have library file names that we accept as a second config.

soumiiow · 2025-02-19T20:06:34Z

@mohsaka @aditi-pandit @PingLiuPing
appreciate the perspective and i see the value in adding the config.
that being said, other than the security vulnerabilities, I think another issue I can think of is having the config with a typo in the filename/path/regex with error handling for that scenario, the user would have to fix the config and restart coordinator and worker for the shared library to be kicked in.

Also for ease of use, i was also thinking, in the absence of a config, we can by default scan in all .dylib/.so files so it's not a requirement for them to put a config in if users just want to plug in and go. thoughts on that?

mohsaka · 2025-02-19T20:10:43Z

@mohsaka @aditi-pandit @PingLiuPing appreciate the perspective and i see the value in adding the config. that being said, other than the security vulnerabilities, I think another issue I can think of is having the config can still crash the worker which is what we would like to avoid in the first place if theres a typo or mismatch in how the user enters the filename/path/regex. Even with error handling for that scenario, the user would have to fix the config and restart coordinator and worker for the shared library to be kicked in.

Also for ease of use, i was also thinking, in the absence of a config, we can by default scan in all .dylib/.so files so it's not a requirement for them to put a config in if users just want to plug in and go. thoughts on that?

I don't think it would crash the worker, we would just not load the library if it didn't match the filepath/regex/etc.

soumiiow · 2025-02-19T20:23:11Z

@mohsaka good point, looking through the velox implementation using dylib, dlerror() would catch the error and give a VELOX_USER_FAIL.
We would still have to restart the coordinator and workers if we want the file to go in so it would be disruptive in that way

soumiiow · 2025-02-24T22:34:08Z

Hey @pedroerp! I was looking through remote_function_server.json to design the config for the dylib changes to be simple to use and similar to the remote fn registrations so as to make for a seamless user experience.

I was curious as to the intended purpose of the schema object for the remote function registrations.

In prestissimo, when we use a prefix/namespace like presto.default, is the equivalent for remote function registrations that "default" would be the schema and that "presto" would be the prefix?

also, i notice that in PrestoServer.cpp, prefix is set to systemConfig->remoteFunctionServerCatalogName(). For remote functions, does this imply that the prefix name would be set for all remote fns and only the schema name would change?

PingLiuPing · 2025-02-24T22:39:19Z

presto-native-execution/presto_cpp/main/dynamic_registry/CMakeLists.txt

+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+add_subdirectory(examples)


I think there is potential issues with regard to upgrading.
You need to fix this by changing some script to trigger the building of the dynamic library when upgrading (from customer perspective). This gurrantee the ABI compatibility in case of major compiler upgrade.
Well this requires the source code should be under some specific directory of presto installation path.

Discussed offline about ABI compatibility issues and came up with documentation changes to alert the users to rebuild shared libraries upon upgrades manually.
presto-docs/src/main/sphinx/presto_cpp/plugin.rst

soumiiow · 2025-03-20T09:28:56Z

Updated the RFC with the discussions regarding adding a Json config, function validation, customizable entrypoint, and signal handling so as to not crash the worker here prestodb/rfcs#24. please take a look!

steveburnett

Thanks for the doc! This will be great to have in the doc. I have some suggestions for conciseness and readability, and a couple of questions for you to consider. Let me know what you think, please!

presto-docs/src/main/sphinx/presto_cpp/plugin/function_plugin.rst

steveburnett · 2025-03-27T13:37:41Z

presto-docs/src/main/sphinx/presto_cpp/plugin/function_plugin.rst

+      add_library(name_of_dynamic_fn SHARED TestFunction.cpp)
+      target_link_libraries(name_of_dynamic_fn PRIVATE fmt::fmt Folly::folly gflags::gflags)
+
+3. Place your shared libraries in the plugin directory. The path to this directory needs to be the same as ``plugin.dir`` property set in :doc:`../plugin`.


Suggested change

3. Place your shared libraries in the plugin directory. The path to this directory needs to be the same as ``plugin.dir`` property set in :doc:`../plugin`.

3. Place your shared libraries in the plugin directory. The path to this directory must be the same as the ``plugin.dir`` property set in :doc:`../plugin`.

steveburnett · 2025-03-27T13:43:39Z

presto-native-execution/presto_cpp/main/dynamic_registry/README.md

@@ -0,0 +1 @@
+Read [here](https://prestodb.io/docs/current/presto-cpp/plugin.html) on how to use the Dynamic Library Loader.


Suggested change

Read [here](https://prestodb.io/docs/current/presto-cpp/plugin.html) on how to use the Dynamic Library Loader.

See [Function Plugin](https://prestodb.io/docs/current/presto-cpp/plugin.html) on how to use the Dynamic Library Loader.

Thanks for addressing the other reviewer's comment by using link to your doc instead of duplicating the content! (If the same text is in two different places then it has to be maintained and updated as twice the work, and errors are likelier to happen.)

When creating a link, here doesn't tell the reader what to expect to find when they open the link. A good practice is to use the title of the destination page in the link, so the reader knows where they are going before they start. (Also, sometimes the title can save the reader time because they might decide this isn't what they need right now.)

steveburnett · 2025-03-27T13:47:16Z

presto-docs/src/main/sphinx/presto_cpp/plugin.rst

+Presto C++ Plugins
+*******************
+
+This chapter outlines the plugins in Presto C++ that are available for various use cases such as to load User Defined Functions (UDFs), connectors, or types.


Suggested change

This chapter outlines the plugins in Presto C++ that are available for various use cases such as to load User Defined Functions (UDFs), connectors, or types.

This page lists the plugins in Presto C++ that are available for various use cases such as to load User Defined Functions (UDFs), connectors, or types, and describes the setup needed to use these plugins.

steveburnett · 2025-03-27T13:51:46Z

presto-docs/src/main/sphinx/presto_cpp/plugin.rst

+
+2. Create a Json configuration file where you will capture information on the shared libraries you wish to load dynamically. 
+
+3. Set the ``plugin.dir`` property to the path of the ``plugins`` directory in the ``config.properties`` file of each of your workers.


Suggested change

3. Set the ``plugin.dir`` property to the path of the ``plugins`` directory in the ``config.properties`` file of each of your workers.

3. Set the ``plugin.dir`` property to the path of the ``plugins`` directory in the ``config.properties`` file of each worker.

steveburnett · 2025-03-27T13:54:54Z

presto-docs/src/main/sphinx/presto_cpp/plugin.rst

+
+2. Create a Json configuration file where you will capture information on the shared libraries you wish to load dynamically. 
+
+3. Set the ``plugin.dir`` property to the path of the ``plugins`` directory in the ``config.properties`` file of each of your workers.


Instead of my two suggestions for lines 20-22, you could consider combining these two as substeps of Step 3. Here's a screenshot of my trying this in a local build.

steveburnett · 2025-03-27T13:55:11Z

presto-docs/src/main/sphinx/presto_cpp/plugin.rst

+
+1. Place the plugin shared libraries in the ``plugins`` directory.
+
+2. Create a Json configuration file where you will capture information on the shared libraries you wish to load dynamically. 


Suggested change

2. Create a Json configuration file where you will capture information on the shared libraries you wish to load dynamically.

2. Create a Json configuration file to capture information on the shared libraries you wish to load dynamically.

Where should this file be created? Can you give an example?

So this file can technically be created anywhere and my current design does not ask for a particular path to make it easier from a deployment perspective. Theres a chance that the person creating the shared library is not the same as the person creating the config. As long as the user gives the full path for the config file it should get read.

So i want to discuss here first if you see value in putting that information into the docs?

Those are excellent points! I agree now that there's no need for it here. Thanks!

steveburnett · 2025-03-27T13:58:17Z

presto-docs/src/main/sphinx/presto_cpp/plugin.rst

+
+5. Start or restart the coordinator and workers to pick up any placed libraries.
+
+Note: to avoid issues with ABI compatibility, we strongly recommend recompiling all shared library plugins during OS and presto version upgrades.


Suggested change

Note: to avoid issues with ABI compatibility, we strongly recommend recompiling all shared library plugins during OS and presto version upgrades.

Note: To avoid issues with ABI compatibility, it is strongly recommended to recompile all shared library plugins during OS and Presto version upgrades.

presto-docs/src/main/sphinx/presto_cpp/plugin/function_plugin.rst

steveburnett · 2025-03-27T14:03:54Z

presto-docs/src/main/sphinx/presto_cpp/plugin/function_plugin.rst

+   * ``nameSpace``: optional field. omitting this field gives the function the default namespace similarly to the function registeration. must match the namespace of the function.  
+   * ``docString``, ``routineCharacteristics``, ``functionKind``: collected for checking metadata.
+
+5. Set the ``plugin.config`` property to the path of Json config. Instructions to set the property in :doc:`../plugin`.


Suggested change

5. Set the ``plugin.config`` property to the path of Json config. Instructions to set the property in :doc:`../plugin`.

5. Set the ``plugin.config`` property to the path of the Json configuration file created in Step 4. See :doc:`../plugin`.

aditi-pandit

@soumiiow : Did a quick pass of your code. Its overall a good design for the first-cut implementation.

aditi-pandit · 2025-03-31T19:56:26Z

presto-native-execution/presto_cpp/main/PrestoServer.cpp

+    const fs::path path(systemConfig->pluginDir());
+    PRESTO_STARTUP_LOG(INFO) << "Dynamic library loading path: " << path;
+    std::error_code ec;
+    if (fs::is_directory(path, ec)) {


It might be simpler to check if there is a validator config and ensure its non-empty before checking the pluginDir path at all.

aditi-pandit · 2025-03-31T19:57:52Z

presto-native-execution/presto_cpp/main/JsonSignatureParser.h

+///      "my_function": [
+///        {
+///          "outputType": "integer",
+///         "entrypoint": "nameOfRegistryFnCall",


Fix the formating of this line.

aditi-pandit · 2025-03-31T20:05:21Z

presto-native-execution/presto_cpp/main/dynamic_registry/DynamicFunctionConfigRegisterer.h

+  int64_t compareConfigWithRegisteredFunctionSignatures(
+      facebook::velox::FunctionSignatureMap fnSignaturesBefore);
+
+  std::


Add a shorthand for
std::unordered_map<std::string, std::vectorvelox::exec::FunctionSignaturePtr> with "using" statement.

aditi-pandit · 2025-03-31T20:07:31Z

presto-native-execution/presto_cpp/main/PrestoServer.cpp

+        DynamicLibraryValidator dVal(configPath, systemConfig->pluginDir());
+        auto filenameAndEntrypointMap = dVal.getEntrypointMap();
+        auto registeredFnSignaturesBefore = velox::getFunctionSignatures();
+        for (const auto& entryPointItr : filenameAndEntrypointMap) {


All this logic can be moved into the DynamicLibraryValidator class (maybe call it DynamicLibraryLoader instead).

The code here would only read the config entries for plugin.dir and dynamiclibraryvalidator and pass them to DynamicLibraryLoader.

aditi-pandit · 2025-03-31T20:09:51Z

presto-native-execution/presto_cpp/main/tests/JsonSignatureParserTest.cpp

@@ -197,5 +197,52 @@ TEST_F(JsonSignatureParserTest, multiple) {
  EXPECT_EQ(signature1->argumentTypes()[1].baseName(), "varchar");
 }

+TEST_F(JsonSignatureParserTest, dynamic) {
+  auto input = R"(


Please add a function which uses a complex type for either paramType or outputType

aditi-pandit · 2025-03-31T20:16:40Z

presto-native-execution/presto_cpp/main/tests/DynamicFunctionRegistererTest.cpp

+    }
+  })";
+
+  // Emulate user provided config file.


There is a some repetition of code between this and the test below. Can you make common functions for reuse ?

aditi-pandit · 2025-03-31T20:18:26Z

...ynamic_registry/examples/function_registerations_non_default/MyDynamicNonDefaultFunction.cpp

Is there a need to separate the directories for this file from the previous directory ?

If yes, then it might be worth trying a recursive directory scenario as well.

aditi-pandit · 2025-03-31T20:19:35Z

...esto_cpp/main/dynamic_registry/examples/function_registerations/MyDynamicVarcharFunction.cpp

Please remove the usage of "My" in the naming of these files.

steveburnett

Thanks for the updated doc! Just a couple of small nits.

steveburnett · 2025-04-08T18:25:45Z

presto-docs/src/main/sphinx/presto_cpp/plugin/function_plugin.rst

+
+   Note: the ``int64_t`` return type, ``registryTest`` registry symbol name can be changed as needed. For more examples, see the `examples <https://github.com/prestodb/presto/tree/master/presto-native-execution/main/dynamic_registry/examples>`_.
+
+2. Create a shared library which may be made using CMakeLists.txt like the following:


Suggested change

2. Create a shared library which may be made using CMakeLists.txt like the following:

2. Create a shared library which may be made using ``CMakeLists.txt`` like the following:

steveburnett · 2025-04-08T18:26:14Z

presto-docs/src/main/sphinx/presto_cpp/plugin/function_plugin.rst

+
+3. Place your shared libraries in the plugin directory. The path to this directory must be the same as the ``plugin.dir`` property set in :doc:`../plugin`.
+
+4. Create a Json configuration file in the same format as below:


Suggested change

4. Create a Json configuration file in the same format as below:

4. Create a JSON configuration file in the same format as below:

soumiiow self-assigned this Jan 7, 2025

soumiiow force-pushed the dylib_new branch from 58ce37c to aedfb72 Compare January 7, 2025 17:16

soumiiow marked this pull request as ready for review January 7, 2025 18:04

soumiiow requested a review from a team as a code owner January 7, 2025 18:04

tdcmeehan self-assigned this Jan 8, 2025

soumiiow mentioned this pull request Jan 14, 2025

presto requires namespaces when using function registration #24363

Open

tdcmeehan added the from:IBM PR from IBM label Jan 30, 2025

prestodb-ci requested review from a team, pdabre12 and psnv03 and removed request for a team January 30, 2025 18:47

czentgr reviewed Feb 3, 2025

View reviewed changes

soumiiow force-pushed the dylib_new branch from aedfb72 to 9cde5a4 Compare February 3, 2025 21:23

soumiiow requested review from steveburnett and a team as code owners February 3, 2025 21:23

soumiiow requested a review from presto-oss February 3, 2025 21:23

steveburnett requested changes Feb 4, 2025

View reviewed changes

soumiiow force-pushed the dylib_new branch from 9cde5a4 to 689b24e Compare February 11, 2025 09:31

soumiiow requested a review from elharo as a code owner February 11, 2025 09:31

soumiiow requested a review from steveburnett February 11, 2025 09:33

soumiiow force-pushed the dylib_new branch from 689b24e to 6cb96e7 Compare February 11, 2025 10:00

steveburnett requested changes Feb 11, 2025

View reviewed changes

soumiiow force-pushed the dylib_new branch from 6cb96e7 to 0ab37d6 Compare February 11, 2025 22:20

steveburnett requested changes Feb 11, 2025

View reviewed changes

presto-docs/src/main/sphinx/presto_cpp/plugin/function_plugin.rst Outdated Show resolved Hide resolved

presto-docs/src/main/sphinx/presto_cpp/plugin/function_plugin.rst Outdated Show resolved Hide resolved

soumiiow force-pushed the dylib_new branch from 0ab37d6 to d8b7dc2 Compare February 12, 2025 18:51

soumiiow requested a review from steveburnett February 12, 2025 18:52

steveburnett requested changes Feb 12, 2025

View reviewed changes

presto-docs/src/main/sphinx/presto_cpp/plugin/function_plugin.rst Outdated Show resolved Hide resolved

soumiiow force-pushed the dylib_new branch from d8b7dc2 to 27bd55e Compare February 12, 2025 19:15

aditi-pandit changed the title ~~Dynamically Linked Library in Presto CPP~~ [native] Dynamically Linked Library in Presto CPP Feb 13, 2025

aditi-pandit reviewed Feb 14, 2025

View reviewed changes

soumiiow dismissed steveburnett’s stale review via 9d94f4f February 19, 2025 00:46

soumiiow force-pushed the dylib_new branch 2 times, most recently from 9d94f4f to ec86c3d Compare February 19, 2025 01:00

soumiiow requested review from czentgr and aditi-pandit February 19, 2025 01:11

PingLiuPing reviewed Feb 19, 2025

View reviewed changes

soumiiow force-pushed the dylib_new branch from ec86c3d to f96893d Compare February 19, 2025 22:14

soumiiow requested a review from PingLiuPing February 19, 2025 22:19

PingLiuPing reviewed Feb 24, 2025

View reviewed changes

soumiiow force-pushed the dylib_new branch 2 times, most recently from a8d2204 to 173ecc9 Compare March 27, 2025 01:16

soumiiow requested review from steveburnett and PingLiuPing March 27, 2025 01:18

steveburnett requested changes Mar 27, 2025

View reviewed changes

aditi-pandit reviewed Mar 31, 2025

View reviewed changes

soumiiow force-pushed the dylib_new branch from 173ecc9 to 94d57ba Compare April 7, 2025 22:03

addressed comments on new changes with config based validation + tests

9ef0b63

soumiiow force-pushed the dylib_new branch from 94d57ba to 9ef0b63 Compare April 7, 2025 22:36

soumiiow requested review from steveburnett and aditi-pandit April 8, 2025 00:07

steveburnett requested changes Apr 8, 2025

View reviewed changes

	# Dynamic Loading of Presto Cpp Extensions
	# Dynamic Loading of Presto CPP Extensions

		@@ -0,0 +1,17 @@
		# Dynamic Loading of Presto Cpp Extensions
		This library adds the ability to load User Defined Functions (UDFs), connectors, or types without having to fork and build Prestissimo, through the use of shared libraries that a Prestissimo worker can access. These are to be loaded on launch of the Presto server. The Presto server searches for any .so or .dylib files and loads them using this library.

	For dynamically loaded function registration, the format followed is mirrored of that of built-in function registration with some noted differences. Using [MyDynamicFunction.cpp](examples/MyDynamicFunction.cpp) as an example, the function uses the extern "C" keyword to protect against name mangling. A registry() function call is also necessary here.
	For dynamically loaded function registration, the format is similar to that of built-in function registration, with some noted differences. Using [MyDynamicFunction.cpp](examples/MyDynamicFunction.cpp) as an example, the function uses the extern "C" keyword to protect against name mangling. A registry() function call is also necessary here.

	4. When the worker or the sidecar process starts, it will scan the plugin directory and attempt to dynamically load all shared libraries
	When the worker or the sidecar process starts, it scans the plugin directory and attempts to dynamically load all shared libraries.

	3. Place your shared libraries in the plugin directory. The path to this directory needs to be the same as ``plugin.dir`` property set in :doc:`../plugin`.
	3. Place your shared libraries in the plugin directory. The path to this directory must be the same as the ``plugin.dir`` property set in :doc:`../plugin`.

		@@ -0,0 +1 @@
		Read [here](https://prestodb.io/docs/current/presto-cpp/plugin.html) on how to use the Dynamic Library Loader.

	Read [here](https://prestodb.io/docs/current/presto-cpp/plugin.html) on how to use the Dynamic Library Loader.
	See [Function Plugin](https://prestodb.io/docs/current/presto-cpp/plugin.html) on how to use the Dynamic Library Loader.

	This chapter outlines the plugins in Presto C++ that are available for various use cases such as to load User Defined Functions (UDFs), connectors, or types.
	This page lists the plugins in Presto C++ that are available for various use cases such as to load User Defined Functions (UDFs), connectors, or types, and describes the setup needed to use these plugins.


		2. Create a Json configuration file where you will capture information on the shared libraries you wish to load dynamically.

		3. Set the ``plugin.dir`` property to the path of the ``plugins`` directory in the ``config.properties`` file of each of your workers.


		1. Place the plugin shared libraries in the ``plugins`` directory.

		2. Create a Json configuration file where you will capture information on the shared libraries you wish to load dynamically.

	2. Create a Json configuration file where you will capture information on the shared libraries you wish to load dynamically.
	2. Create a Json configuration file to capture information on the shared libraries you wish to load dynamically.


		5. Start or restart the coordinator and workers to pick up any placed libraries.

		Note: to avoid issues with ABI compatibility, we strongly recommend recompiling all shared library plugins during OS and presto version upgrades.

	5. Set the ``plugin.config`` property to the path of Json config. Instructions to set the property in :doc:`../plugin`.
	5. Set the ``plugin.config`` property to the path of the Json configuration file created in Step 4. See :doc:`../plugin`.


		Note: the ``int64_t`` return type, ``registryTest`` registry symbol name can be changed as needed. For more examples, see the `examples <https://github.com/prestodb/presto/tree/master/presto-native-execution/main/dynamic_registry/examples>`_.

		2. Create a shared library which may be made using CMakeLists.txt like the following:


		3. Place your shared libraries in the plugin directory. The path to this directory must be the same as the ``plugin.dir`` property set in :doc:`../plugin`.

		4. Create a Json configuration file in the same format as below:

[native] Dynamically Linked Library in Presto CPP #24330

Are you sure you want to change the base?

[native] Dynamically Linked Library in Presto CPP #24330

Conversation

soumiiow commented Jan 7, 2025

Description

Motivation and Context

Impact

Test Plan

Contributor checklist

Release Notes

linux-foundation-easycla bot commented Jan 7, 2025 • edited Loading

czentgr left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

steveburnett left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

steveburnett left a comment

Choose a reason for hiding this comment

soumiiow commented Feb 11, 2025

steveburnett left a comment

Choose a reason for hiding this comment

steveburnett left a comment

Choose a reason for hiding this comment

aditi-pandit left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mohsaka commented Feb 17, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mohsaka commented Feb 19, 2025 • edited Loading

soumiiow commented Feb 19, 2025 • edited Loading

mohsaka commented Feb 19, 2025

soumiiow commented Feb 19, 2025

soumiiow commented Feb 24, 2025

Choose a reason for hiding this comment

Choose a reason for hiding this comment

soumiiow commented Mar 20, 2025

steveburnett left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

soumiiow Apr 1, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

aditi-pandit left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

steveburnett left a comment

Choose a reason for hiding this comment

linux-foundation-easycla bot commented Jan 7, 2025 •

edited

Loading

mohsaka commented Feb 17, 2025 •

edited

Loading

mohsaka commented Feb 19, 2025 •

edited

Loading

soumiiow commented Feb 19, 2025 •

edited

Loading

soumiiow Apr 1, 2025 •

edited

Loading