Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

embed dll loader #120

Merged
merged 3 commits into from
Jan 27, 2025
Merged

embed dll loader #120

merged 3 commits into from
Jan 27, 2025

Conversation

cocoa-xu
Copy link
Member

@cocoa-xu cocoa-xu merged commit 8ffc75f into main Jan 27, 2025
3 checks passed
@cocoa-xu cocoa-xu deleted the cx-embed-dll-loader branch January 27, 2025 10:34
@josevalim
Copy link
Member

Amazing @cocoa-xu! Agreed with the fix :)

@cocoa-xu
Copy link
Member Author

Amazing @cocoa-xu! Agreed with the fix :)

Although the error message from Windows was Unspecified error so I'm not quite sure if this PR fully addressed the mention issue...

@DnOberon I shipped v0.7.4 which now embeds the dll loader for Windows. Please let me know if that works for you. And if not, could you please also include some steps for me to reproduce?

@DnOberon
Copy link
Contributor

Sound excellent @cocoa-xu, will do

@DnOberon
Copy link
Contributor

DnOberon commented Jan 27, 2025

Alright @cocoa-xu the plot thickens - what information do you need from me to reproduce?

Windows 10 - added Visual Studio Code Build Tools 2022 and the C++ build section. Running the Developer Powershell. Fresh project using the latest adbc .

image

Seems like the files compiled or downloaded successfully though?
image

@cocoa-xu
Copy link
Member Author

@DnOberon Okay, v0.7.5 should work now :)

@DnOberon
Copy link
Contributor

Hmm the error doesn't seem as big - but still getting one

@DnOberon
Copy link
Contributor

image

@cocoa-xu
Copy link
Member Author

Hi @DnOberon could you please share the commands you were using that can lead to this result?

@DnOberon
Copy link
Contributor

DnOberon commented Jan 27, 2025 via email

@cocoa-xu
Copy link
Member Author

Sorry but I couldn't reproduce the same error... It would be really helpful if you could provide more detailed commands that you were using

C:\Users\cocoa>git clone https://github.com/elixir-explorer/adbc.git
Cloning into 'adbc'...
remote: Enumerating objects: 3365, done.
remote: Counting objects: 100% (755/755), done.
remote: Compressing objects: 100% (258/258), done.
remote: Total 3365 (delta 645), reused 512 (delta 492), pack-reused 2610 (from 2)
Receiving objects: 100% (3365/3365), 4.76 MiB | 10.50 MiB/s, done.
Resolving deltas: 100% (2058/2058), done.

C:\Users\cocoa>cd adbc

C:\Users\cocoa\adbc>mix deps.get
Resolving Hex dependencies...
Resolution completed in 0.078s
Unchanged:
  castore 1.0.6
  cc_precompiler 0.1.10
  decimal 2.3.0
  earmark_parser 1.4.42
  elixir_make 0.8.3
  ex_doc 0.36.1
  makeup 1.2.1
  makeup_elixir 1.0.1
  makeup_erlang 1.0.1
  nimble_parsec 1.4.1
* Getting cc_precompiler (Hex package)
* Getting elixir_make (Hex package)
* Getting castore (Hex package)
* Getting decimal (Hex package)
* Getting ex_doc (Hex package)
* Getting earmark_parser (Hex package)
* Getting makeup_elixir (Hex package)
* Getting makeup_erlang (Hex package)
* Getting makeup (Hex package)
* Getting nimble_parsec (Hex package)

C:\Users\cocoa\adbc>mix deps.compile
==> decimal
Compiling 4 files (.ex)
Generated decimal app
==> castore
Compiling 1 file (.ex)
Generated castore app
==> elixir_make
Compiling 6 files (.ex)
Generated elixir_make app
==> cc_precompiler
Compiling 3 files (.ex)
Generated cc_precompiler app

C:\Users\cocoa\adbc>mix compile
Error happened while installing adbc from precompiled binary: "missing checksum.exs file".

Attempting to compile adbc from source...

Microsoft (R) Program Maintenance Utility Version 14.42.34436.0
Copyright (C) Microsoft Corporation.  All rights reserved.

Not searching for unused variables given on the command line.
CMake Warning:
  Ignoring empty string ("") provided on the command line.


-- The C compiler identification is MSVC 19.42.34436.0
-- The CXX compiler identification is MSVC 19.42.34436.0
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working C compiler: C:/Program Files/Microsoft Visual Studio/2022/Community/VC/Tools/MSVC/14.42.34433/bin/Hostx64/x64/cl.exe - skipped
-- Detecting C compile features
-- Detecting C compile features - done
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Check for working CXX compiler: C:/Program Files/Microsoft Visual Studio/2022/Community/VC/Tools/MSVC/14.42.34433/bin/Hostx64/x64/cl.exe - skipped
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Performing Test CXX_LINKER_SUPPORTS_VERSION_SCRIPT
-- Performing Test CXX_LINKER_SUPPORTS_VERSION_SCRIPT - Failed
-- {fmt} version: 10.2.2
-- Build type: Release
-- ---------------------------------------------------------------------
-- ADBC version: 1.4.0
--
-- Build configuration summary:
--   CMake version: 3.29.5-msvc4
--   Generator: NMake Makefiles
--   Build type: Release
--   Source directory: C:/Users/cocoa/adbc/3rd_party/apache-arrow-adbc/c
--   Install prefix: C:/Users/cocoa/adbc/_build/dev/lib/adbc/priv
--
-- Compile and link options:
--
--   ADBC_BUILD_WARNING_LEVEL=PRODUCTION [default=""]
--       CHECKIN to enable Werror, PRODUCTION otherwise
--   ADBC_CXXFLAGS="" [default=""]
--       Compiler flags to append when compiling ADBC C++ libraries
--   ADBC_GO_BUILD_TAGS="" [default=""]
--       Build tags to append when compiling ADBC Go libraries
--   ADBC_BUILD_STATIC=OFF [default=ON]
--       Build static libraries
--   ADBC_BUILD_SHARED=ON [default=ON]
--       Build shared libraries
--   ADBC_PACKAGE_KIND="" [default=""]
--       Arbitrary string that identifies the kind of package
--       (for informational purposes)
--   ADBC_GIT_ID=b2c9a1498ff9dfb4d81dd986c78b7924fd0b032f [default=""]
--       The Arrow git commit id (if any)
--   ADBC_GIT_DESCRIPTION=v0.7.5 [default=""]
--       The Arrow git commit description (if any)
--   ADBC_NO_DEPRECATED_API=OFF [default=OFF]
--       Exclude deprecated APIs from build
--   ADBC_USE_CCACHE=ON [default=ON]
--       Use ccache when compiling (if available)
--   ADBC_USE_PRECOMPILED_HEADERS=OFF [default=OFF]
--       Use precompiled headers when compiling
--   ADBC_ARMV8_ARCH=armv8-a [default=armv8-a|armv8-a+crc+crypto]
--       Arm64 arch and extensions
--   ADBC_ALTIVEC=ON [default=ON]
--       Build with Altivec if compiler has support
--   ADBC_RPATH_ORIGIN=OFF [default=OFF]
--       Build Arrow libraries with RATH set to $ORIGIN
--   ADBC_INSTALL_NAME_RPATH=ON [default=ON]
--       Build Arrow libraries with install_name set to @rpath
--   ADBC_GGDB_DEBUG=ON [default=ON]
--       Pass -ggdb flag to debug builds
--
-- Test and benchmark options:
--
--   ADBC_BUILD_EXAMPLES=OFF [default=OFF]
--       Build the Arrow examples
--   ADBC_BUILD_TESTS=OFF [default=OFF]
--       Build the Arrow googletest unit tests
--   ADBC_BUILD_INTEGRATION=OFF [default=OFF]
--       Build the Arrow integration test executables
--   ADBC_BUILD_BENCHMARKS=OFF [default=OFF]
--       Build the Arrow micro benchmarks
--   ADBC_TEST_LINKAGE=shared [default=shared|static]
--       Linkage of Arrow libraries with unit tests executables.
--
-- Lint options:
--
--   ADBC_GENERATE_COVERAGE=OFF [default=OFF]
--       Build with C++ code coverage enabled
--
-- Checks options:
--
--   ADBC_TEST_MEMCHECK=OFF [default=OFF]
--       Run the test suite using valgrind --tool=memcheck
--   ADBC_USE_ASAN=OFF [default=OFF]
--       Enable Address Sanitizer checks
--   ADBC_USE_TSAN=OFF [default=OFF]
--       Enable Thread Sanitizer checks
--   ADBC_USE_UBSAN=OFF [default=OFF]
--       Enable Undefined Behavior sanitizer checks
--
-- Thirdparty toolchain options:
--
--   ADBC_DEPENDENCY_SOURCE=BUNDLED [default=AUTO|BUNDLED|SYSTEM|CONDA|VCPKG|BREW]
--       Method to use for acquiring arrow's build dependencies
--
-- Advanced developer options:
--
--
-- Project components options:
--
--   ADBC_DRIVER_FLIGHTSQL=OFF [default=OFF]
--       Build the Flight SQL driver
--   ADBC_DRIVER_MANAGER=ON [default=OFF]
--       Build the driver manager
--   ADBC_DRIVER_POSTGRESQL=OFF [default=OFF]
--       Build the PostgreSQL driver
--   ADBC_DRIVER_SQLITE=OFF [default=OFF]
--       Build the SQLite driver
--   ADBC_DRIVER_SNOWFLAKE=OFF [default=OFF]
--       Build the Snowflake driver
--   ADBC_DRIVER_BIGQUERY=OFF [default=OFF]
--       Build the BigQuery driver
--   ADBC_INTEGRATION_DUCKDB=OFF [default=OFF]
--       Build the test suite for DuckDB
-- Configuring done (3.5s)
-- Generating done (0.2s)
-- Build files have been written to: C:/Users/cocoa/adbc/_build/dev/lib/adbc/cmake_adbc
Warning: NMake does not support parallel builds. Ignoring parallel build command line option.
[  8%] Building C object vendor/nanoarrow/CMakeFiles/nanoarrow.dir/nanoarrow.c.obj
nanoarrow.c
[ 16%] Linking C static library nanoarrow.lib
[ 16%] Built target nanoarrow
[ 25%] Building C object driver/common/CMakeFiles/adbc_driver_common.dir/utils.c.obj
utils.c
[ 33%] Linking C static library adbc_driver_common.lib
[ 33%] Built target adbc_driver_common
[ 41%] Building CXX object vendor/fmt/CMakeFiles/fmt.dir/src/format.cc.obj
format.cc
[ 50%] Building CXX object vendor/fmt/CMakeFiles/fmt.dir/src/os.cc.obj
os.cc
[ 58%] Linking CXX static library fmt.lib
[ 58%] Built target fmt
[ 66%] Building CXX object driver/framework/CMakeFiles/adbc_driver_framework.dir/objects.cc.obj
objects.cc
[ 75%] Building CXX object driver/framework/CMakeFiles/adbc_driver_framework.dir/utility.cc.obj
utility.cc
[ 83%] Linking CXX static library adbc_driver_framework.lib
[ 83%] Built target adbc_driver_framework
[ 91%] Building CXX object driver_manager/CMakeFiles/adbc_driver_manager_shared.dir/adbc_driver_manager.cc.obj
adbc_driver_manager.cc
C:\Users\cocoa\adbc\3rd_party\apache-arrow-adbc\c\driver_manager\adbc_driver_manager.cc(678): warning C4244: '=': conversion from 'int' to 'char', possible loss of data
[100%] Linking CXX shared library adbc_driver_manager.dll
[100%] Built target adbc_driver_manager_shared
Install the project...
-- Install configuration: "Release"
-- Installing: C:/Users/cocoa/adbc/_build/dev/lib/adbc/priv/include/adbc_driver_manager.h
-- Installing: C:/Users/cocoa/adbc/_build/dev/lib/adbc/priv/include/arrow-adbc/adbc_driver_manager.h
-- Installing: C:/Users/cocoa/adbc/_build/dev/lib/adbc/priv/lib/adbc_driver_manager.lib
-- Installing: C:/Users/cocoa/adbc/_build/dev/lib/adbc/priv/bin/adbc_driver_manager.dll
-- Installing: C:/Users/cocoa/adbc/_build/dev/lib/adbc/priv/lib/cmake/AdbcDriverManager/AdbcDriverManagerConfig.cmake
-- Installing: C:/Users/cocoa/adbc/_build/dev/lib/adbc/priv/lib/cmake/AdbcDriverManager/AdbcDriverManagerConfigVersion.cmake
-- Installing: C:/Users/cocoa/adbc/_build/dev/lib/adbc/priv/lib/cmake/AdbcDriverManager/AdbcDriverManagerTargets.cmake
-- Installing: C:/Users/cocoa/adbc/_build/dev/lib/adbc/priv/lib/cmake/AdbcDriverManager/AdbcDriverManagerTargets-release.cmake
-- Installing: C:/Users/cocoa/adbc/_build/dev/lib/adbc/priv/lib/pkgconfig/adbc-driver-manager.pc
-- Installing: C:/Users/cocoa/adbc/_build/dev/lib/adbc/priv/include/adbc.h
-- Installing: C:/Users/cocoa/adbc/_build/dev/lib/adbc/priv/include/arrow-adbc/adbc.h
Not searching for unused variables given on the command line.
CMake Warning:
  Ignoring empty string ("") provided on the command line.


-- The C compiler identification is MSVC 19.42.34436.0
-- The CXX compiler identification is MSVC 19.42.34436.0
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working C compiler: C:/Program Files/Microsoft Visual Studio/2022/Community/VC/Tools/MSVC/14.42.34433/bin/Hostx64/x64/cl.exe - skipped
-- Detecting C compile features
-- Detecting C compile features - done
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Check for working CXX compiler: C:/Program Files/Microsoft Visual Studio/2022/Community/VC/Tools/MSVC/14.42.34433/bin/Hostx64/x64/cl.exe - skipped
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Using PRIV_DIR: c:/Users/cocoa/adbc/_build/dev/lib/adbc/priv
-- Using ERTS_INCLUDE_DIR: c:/Program Files/Erlang OTP/erts-15.2.1/include
-- CMAKE_TOOLCHAIN_FILE:
-- Configuring done (2.3s)
-- Generating done (0.1s)
-- Build files have been written to: C:/Users/cocoa/adbc/_build/dev/lib/adbc/cmake_adbc_nif
[ 14%] Building C object 3rd_party/apache-arrow-adbc/c/vendor/nanoarrow/CMakeFiles/nanoarrow.dir/nanoarrow.c.obj
nanoarrow.c
[ 28%] Linking C static library nanoarrow.lib
[ 28%] Built target nanoarrow
[ 42%] Building CXX object CMakeFiles/adbc_nif.dir/c_src/nif_utils.cpp.obj
nif_utils.cpp
[ 57%] Building CXX object CMakeFiles/adbc_nif.dir/c_src/adbc_nif.cpp.obj
adbc_nif.cpp
[ 71%] Linking CXX shared library adbc_nif.dll
[ 71%] Built target adbc_nif
[ 85%] Building CXX object CMakeFiles/adbc_dll_loader.dir/c_src/dll_loader/adbc_dll_loader.cpp.obj
adbc_dll_loader.cpp
[100%] Linking CXX shared library adbc_dll_loader.dll
[100%] Built target adbc_dll_loader
-- Installing: c:/Users/cocoa/adbc/_build/dev/lib/adbc/priv/adbc_nif.dll
-- Installing: c:/Users/cocoa/adbc/_build/dev/lib/adbc/priv/adbc_dll_loader.lib
-- Installing: c:/Users/cocoa/adbc/_build/dev/lib/adbc/priv/adbc_dll_loader.dll
Compiling 10 files (.ex)
Generated adbc app

C:\Users\cocoa\adbc>

@cocoa-xu cocoa-xu added the question Further information is requested label Jan 27, 2025
@DnOberon
Copy link
Contributor

Will do. Give me a bit and I'll make a fresh project and turn debug on :)

@DnOberon
Copy link
Contributor

DnOberon commented Jan 27, 2025

PS C:\Users\family\datum-windows> mix deps.compile adbc
-> Running mix CLI
-> Running mix loadconfig (inside Datum.MixProject)
<- Ran mix loadconfig in 9ms
-> Running mix deps.compile adbc (inside Datum.MixProject)
-> Running mix archive.check adbc (inside Datum.MixProject)
<- Ran mix archive.check in 0ms
-> Running mix deps.precompile (inside Datum.MixProject)
<- Ran mix deps.precompile in 0ms
==> adbc
-> Running mix compile --from-mix-deps-compile --no-warnings-as-errors --no-code-path-pruning (inside Adbc.MixProject)
-> Running mix loadpaths --from-mix-deps-compile --no-warnings-as-errors --no-code-path-pruning (inside Adbc.MixProject)
<- Ran mix loadpaths in 1ms
-> Running mix compile.all --from-mix-deps-compile --no-warnings-as-errors --no-code-path-pruning (inside Adbc.MixProject)
-> Running mix compile.yecc --from-mix-deps-compile --no-warnings-as-errors --no-code-path-pruning (inside Adbc.MixProject)
<- Ran mix compile.yecc in 3ms
-> Running mix compile.leex --from-mix-deps-compile --no-warnings-as-errors --no-code-path-pruning (inside Adbc.MixProject)
<- Ran mix compile.leex in 0ms
-> Running mix compile.elixir_make --from-mix-deps-compile --no-warnings-as-errors --no-code-path-pruning (inside Adbc.MixProject)
<- Ran mix compile.elixir_make in 108ms
-> Running mix compile.erlang --from-mix-deps-compile --no-warnings-as-errors --no-code-path-pruning (inside Adbc.MixProject)
<- Ran mix compile.erlang in 0ms
-> Running mix compile.elixir --from-mix-deps-compile --no-warnings-as-errors --no-code-path-pruning (inside Adbc.MixProject)
Compiling 10 files (.ex)
Failed to load nif: {:load_failed, ~c"Failed to load NIF library c:/Users/family/datum-windows/_build/dev/lib/adbc/priv/adbc_nif: 'The specified module could not be found.'"}
<- Ran mix compile.elixir in 1493ms
-> Running mix compile.app --from-mix-deps-compile --no-warnings-as-errors --no-code-path-pruning (inside Adbc.MixProject)
Generated adbc app
<- Ran mix compile.app in 7ms
<- Ran mix compile.all in 1630ms
<- Ran mix compile in 1633ms
==> datum
-> Running mix will_recompile (inside Datum.MixProject)
<- Ran mix will_recompile in 0ms
<- Ran mix deps.compile in 2266ms
<- Ran mix CLI in 2390ms
PS C:\Users\family\datum-windows> cl \version
Microsoft (R) C/C++ Optimizing Compiler Version 19.42.34436 for x86
Copyright (C) Microsoft Corporation.  All rights reserved.

is this helpful? Happens both in existing and fresh project with ADBC as the only dependency.

If I copy the release in manually from the release on Github, it works. So I'm assuming it's something up with my compiling going wrong.

Edit: Oh and I'm using the DuckDB driver

@DnOberon
Copy link
Contributor

DnOberon commented Jan 27, 2025

Now it's not working on MacOS :
image

Verified regression as it works in 0.7.3

@josevalim
Copy link
Member

Yes, I can reproduce the failure on macOS too. Maybe we should release those versions as release candidates, so we don't break existing workflows? For example, Livebook relies on ADBC for some work and it will fetch the latest version by default. :)

@cocoa-xu
Copy link
Member Author

I'm not sure why it needs to be compiled separately...but doing mix deps.compile or mix release should be more than enough

image

@DnOberon
Copy link
Contributor

I did it separately @cocoa-xu so you could more easily see the error. The same error appears on mix compile or mix release

@josevalim
Copy link
Member

It works on macOS again, thanks @cocoa-xu :)

@cocoa-xu
Copy link
Member Author

You mean this message?

Failed to load nif: {:load_failed, ~c"Failed to load NIF library c:/Users/cocoa/mytest/_build/prod/lib/adbc/priv/adbc_nif: 'The specified module could not be found.'"}
image

It can be safely ignored during compile-time.

@DnOberon
Copy link
Contributor

Correct

@DnOberon
Copy link
Contributor

Alright then let me recompile and run on windows on the latest and see if it happens at runtime too. Thanks

@cocoa-xu
Copy link
Member Author

Yes, I can reproduce the failure on macOS too. Maybe we should release those versions as release candidates, so we don't break existing workflows? For example, Livebook relies on ADBC for some work and it will fetch the latest version by default. :)

I really need to setup a dev hex.pm repo for testing..

@josevalim
Copy link
Member

@cocoa-xu you can use mix hex.build to build the package and then unpack it. We do this to test ExDoc:

https://github.com/elixir-lang/ex_doc/blob/main/test/prerelease.sh

Maybe we could have a similar script here?

@cocoa-xu
Copy link
Member Author

you can use mix hex.build to build the package and then unpack it
Maybe we could have a similar script here?

Oh this sounds like a good fit here, I'll look into this later today!

@DnOberon
Copy link
Contributor

DnOberon commented Jan 27, 2025

Seems like I can't ignore the error - because I get that same error on runtime when attempting to open a database connection
image

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
question Further information is requested
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants