[Build] Undefined symbol cross compiling on MacOs for armeabi-v7a

[antonivanov-r]

Describe the issue

Hi,
I'm getting undefined symbol during linking for armeabi-v7a.
arm64-v8a compiles and links fine.

Urgency

v1.22.0 built fine, so it's a regression

Target platform

Android

Build script

command = [
str(onnx_dir / "build.sh"),
"--build_dir", f"{abi_dir}",
"--android_sdk_path", f"{android_home}",
"--android_ndk_path", f"{ndk_home}",
"--android_abi", f"{abi}",
"--minimal_build", "extended",
"--parallel",
"--skip_tests",
"--config=MinSizeRel",
"--disable_exceptions",
"--android_api", f"{api_ver}",
"--android",
"--enable_lto",
"--disable_ml_ops",
"--build_shared_lib",
"--cmake_extra_defines=onnxruntime_BUILD_UNIT_TESTS=OFF"
]

Error / output

[100%] Linking CXX static library libonnxruntime_providers.a
[100%] Built target onnxruntime_providers
[100%] Building CXX object CMakeFiles/onnxruntime.dir/Users/user/Developer/onnxruntime/onnxruntime/onnxruntime/core/platform/android/cxa_demangle.cc.o
[100%] Building C object CMakeFiles/onnxruntime.dir/generated_source.c.o
[100%] Linking CXX shared library libonnxruntime.so
ld.lld: error: undefined symbol: ArmKleidiAI::MlasDynamicQGemmBatch(MLAS_GEMM_DYN_QUANT_SHAPE_PARAMS const&, MLAS_GEMM_DYN_QUANT_DATA_PARAMS const*, unsigned int, onnxruntime::concurrency::ThreadPool*)
>>> referenced by dynamic_quantize_matmul.cc
>>>               lto.tmp:(onnxruntime::contrib::DynamicQuantizeMatMul::Compute(onnxruntime::OpKernelContext*) const)

ld.lld: error: undefined symbol: ArmKleidiAI::MlasDynamicQgemmPackBSize(unsigned int, unsigned int)
>>> referenced by dynamic_quantize_matmul.cc
>>>               lto.tmp:(onnxruntime::contrib::DynamicQuantizeMatMul::PrePack(onnxruntime::Tensor const&, int, std::__ndk1::shared_ptr<onnxruntime::IAllocator>, bool&, onnxruntime::PrePackedWeights*))
>>> referenced by qgemm.cpp
>>>               lto.tmp:(MlasGemmPackBSize(unsigned int, unsigned int, bool, bool))

ld.lld: error: undefined symbol: ArmKleidiAI::MlasDynamicQgemmPackB(unsigned int, unsigned int, signed char const*, float const*, float const*, void*)
>>> referenced by dynamic_quantize_matmul.cc
>>>               lto.tmp:(onnxruntime::contrib::DynamicQuantizeMatMul::PrePack(onnxruntime::Tensor const&, int, std::__ndk1::shared_ptr<onnxruntime::IAllocator>, bool&, onnxruntime::PrePackedWeights*))
clang++: error: linker command failed with exit code 1 (use -v to see invocation)
make[2]: *** [libonnxruntime.so] Error 1
make[1]: *** [CMakeFiles/onnxruntime.dir/all] Error 2

Visual Studio Version

No response

GCC / Compiler Version

No response

[edgchen1]

What onnxruntime version are you building?

A workaround to try is passing --no_kleidiai to build.py.

[antonivanov-r]

@edgchen1
I'm using v1.23.3.

[Colm-in-Arm]

Hello antonivanov-r

armeabi-v7a is a 32bit Arm v7 build. This is checked for in build.py. Can you describe how you're building this please?

As Ed said, adding --no_kleidiai to build.py should resolve this. If you are calling the cmake directly perhaps you could investigate how Donnxruntime_USE_KLEIDIAI is being set to ON? By default it is OFF.

Colm.

[antonivanov-r]

Hey @edgchen1 @Colm-in-Arm
I'm using the build.sh script provided. You can see the toggles I'm using in the problem description. I'll try setting Donnxruntime_USE_KLEIDIAI=OFF

[edgchen1]

I'm using v1.23.3.

assuming that this is v1.23.2 as there is currently no 1.23.3.

I was unable to reproduce the issue at tag v1.23.2. I tried cross-compiling for Android armeabi-v7a on an Intel Mac and it was successful.

% cat arm7_build_issue.opt
--build_dir build/arm7_build_issue
--cmake_generator Ninja
--android
--android_sdk_path ~/Library/Android/sdk
--android_ndk_path ~/Library/Android/sdk/ndk/27.0.12077973
--android_abi armeabi-v7a
--android_api 31
--minimal_build extended
--disable_exceptions
--enable_lto
--disable_ml_ops
--build_shared_lib
--cmake_extra_defines=onnxruntime_BUILD_UNIT_TESTS=OFF

% ./build.sh "@./arm7_build_issue.opt" --config MinSizeRel --update --build --parallel

[sevenc-nanashi]

I'm facing the same issue, on Ubuntu (WSL) with onnxruntime 1.23.2.

[ 94%] Building CXX object CMakeFiles/onnxruntime_test_all.dir/home/sevenc7c/onnxruntime/onnxruntime/test/lora/lora_test.cc.o
[ 94%] Building CXX object CMakeFiles/onnxruntime_test_all.dir/home/sevenc7c/onnxruntime/onnxruntime/test/unittest_main/test_main.cc.o
[ 95%] Linking CXX executable onnxruntime_test_all
/bin/ld: warning: QgemmU8S8KernelAmx.S.o: missing .note.GNU-stack section implies executable stack
/bin/ld: NOTE: This behaviour is deprecated and will be removed in a future version of the linker
/bin/ld: libonnxruntime_mlas.a(qgemm.cpp.o): in function `MlasDynamicQGemmBatch(MLAS_GEMM_DYN_QUANT_SHAPE_PARAMS const&, MLAS_GEMM_DYN_QUANT_DATA_PARAMS const*, unsigned long, onnxruntime::concurrency::ThreadPool*)':
/home/sevenc7c/onnxruntime/onnxruntime/core/mlas/lib/qgemm.cpp:215:(.text._Z21MlasDynamicQGemmBatchRK32MLAS_GEMM_DYN_QUANT_SHAPE_PARAMSPK31MLAS_GEMM_DYN_QUANT_DATA_PARAMSmPN11onnxruntime11concurrency10ThreadPoolE+0x41): undefined reference to `ArmKleidiAI::MlasDynamicQGemmBatch(MLAS_GEMM_DYN_QUANT_SHAPE_PARAMS const&, MLAS_GEMM_DYN_QUANT_DATA_PARAMS const*, unsigned long, onnxruntime::concurrency::ThreadPool*)'
/bin/ld: libonnxruntime_mlas.a(qgemm.cpp.o): in function `MlasDynamicQgemmPackBSize(unsigned long, unsigned long)':
/home/sevenc7c/onnxruntime/onnxruntime/core/mlas/lib/qgemm.cpp:340:(.text._Z25MlasDynamicQgemmPackBSizemm+0x3c): undefined reference to `ArmKleidiAI::MlasDynamicQgemmPackBSize(unsigned long, unsigned long)'
/bin/ld: libonnxruntime_mlas.a(qgemm.cpp.o): in function `MlasDynamicQgemmPackB(unsigned long, unsigned long, signed char const*, float const*, float const*, void*)':
/home/sevenc7c/onnxruntime/onnxruntime/core/mlas/lib/qgemm.cpp:427:(.text._Z21MlasDynamicQgemmPackBmmPKaPKfS2_Pv+0x57): undefined reference to `ArmKleidiAI::MlasDynamicQgemmPackB(unsigned long, unsigned long, signed char const*, float const*, float const*, void*)'
collect2: error: ld returned 1 exit status
gmake[2]: *** [CMakeFiles/onnxruntime_test_all.dir/build.make:5407: onnxruntime_test_all] Error 1
gmake[1]: *** [CMakeFiles/Makefile2:1945: CMakeFiles/onnxruntime_test_all.dir/all] Error 2
gmake: *** [Makefile:146: all] Error 2
Traceback (most recent call last):
  File "/home/sevenc7c/onnxruntime/tools/ci_build/build.py", line 2670, in <module>
    sys.exit(main())
             ^^^^^^
  File "/home/sevenc7c/onnxruntime/tools/ci_build/build.py", line 2571, in main
    build_targets(args, cmake_path, build_dir, configs, num_parallel_jobs, args.target)
  File "/home/sevenc7c/onnxruntime/tools/ci_build/build.py", line 1363, in build_targets
    run_subprocess(cmd_args, env=env)
  File "/home/sevenc7c/onnxruntime/tools/ci_build/build.py", line 148, in run_subprocess
    return run(*args, cwd=cwd, capture_stdout=capture_stdout, shell=shell, env=my_env)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/sevenc7c/onnxruntime/tools/python/util/run.py", line 50, in run
    completed_process = subprocess.run(
                        ^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/subprocess.py", line 571, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['/home/sevenc7c/.local/share/cmvm/current/bin/cmake', '--build', '/home/sevenc7c/onnxruntime/build/Linux/Debug', '--config', 'Debug']' returned non-zero exit status 2.

[edgchen1]

@sevenc-nanashi Can you share instructions on how to reproduce it? Preferably, including the entire build command used.

[sevenc-nanashi]

Here it is:
onnxruntime.log

I ran ./build.sh --clean, then ./build.sh.
I'm at v1.23.2 (a83fc4d)

Also, --no_kleidiai did not work: onnxruntime_no_kleidiai.log
while --cmake_extra_defines onnxruntime_USE_KLEIDIAI=OFF works:

onnxruntime_cmake_def.log

[edgchen1]

It appears that onnxruntime_USE_KLEIDIAI got set somehow for your build directory. Not sure how that happened.

As a side note, subsequent builds in the same build directory will use the old CMake option value that was set, unless it is overridden. It's usually better to troubleshoot these build issues by starting from a completely new build directory. --clean won't actually remove the build directory.

It's good that you found a workaround. If that's sufficient, fine. If not, can you try building from main? Some of the code related to onnxruntime_USE_KLEIDIAI was reworked and it would be interesting to see if it still can be reproduced.

[sevenc-nanashi]

While test itself is failed, I could compile and test was ran on current main (ea55c16). Thanks for the information!

[antonivanov-r]

I was able to build with --cmake_extra_defines onnxruntime_USE_KLEIDIAI=OFF as well.