Download CK library from compute-artifactory and link to Pytorch #2007

akashveramd · 2025-03-26T22:56:17Z

These are the changes made for this PR-

Removed generating and building CK kernels when building pytorch.
Download CK library from compute-artifactory and link with pytorch.
Enabled USE_CK_FLASH_ATTENTION based on USE_FLASH_ATTENTION option.

…eate link target. Enable USE_CK_FLASH_ATTENTION based on USE_FLASH_ATTENTION option.

rocm-repo-management-api · 2025-03-26T23:40:57Z

Jenkins build for cc09d84cab096e6aeca0d3d088b694746f363164 commit finished as FAILURE
Links: Blue Ocean view / Build artifacts

Detected error during Pytorch building:

[7790/7886] Linking CXX shared library lib/libbackend_with_compiler.so
Warning: Unused direct dependencies:
	/var/lib/jenkins/pytorch/build/lib/libtorch.so
	/var/lib/jenkins/pytorch/build/lib/libtorch_hip.so
[7791/7886] Linking CXX executable bin/Dict_test
FAILED: bin/Dict_test 
: && /opt/cache/bin/c++ -D_GLIBCXX_USE_CXX11_ABI=1 -fvisibility-inlines-hidden -DUSE_PTHREADPOOL -DNDEBUG -DUSE_KINETO -DLIBKINETO_NOCUPTI -DLIBKINETO_NOXPUPTI=ON -DUSE_FBGEMM -DUSE_PYTORCH_QNNPACK -DUSE_XNNPACK -DSYMBOLICATE_MOBILE_DEBUG_HANDLE -O2 -fPIC -Wall -Wextra -Werror=return-type -Werror=non-virtual-dtor -Werror=range-loop-construct -Werror=bool-operation -Wnarrowing -Wno-missing-field-initializers -Wno-type-limits -Wno-array-bounds -Wno-unknown-pragmas -Wno-unused-parameter -Wno-strict-overflow -Wno-strict-aliasing -Wno-stringop-overflow -Wsuggest-override -Wno-psabi -Wno-error=old-style-cast -Wno-missing-braces -fdiagnostics-color=always -faligned-new -Wno-unused-but-set-variable -Wno-maybe-uninitialized -fno-math-errno -fno-trapping-math -Werror=format -Wno-stringop-overflow -DHAVE_AVX512_CPU_DEFINITION -DHAVE_AVX2_CPU_DEFINITION -O3 -DNDEBUG -DNDEBUG -rdynamic     -Wl,--no-as-needed caffe2/CMakeFiles/Dict_test.dir/__/aten/src/ATen/test/Dict_test.cpp.o -o bin/Dict_test -L/lib/intel64   -L/lib/intel64_win   -L/lib/win-x64 -Wl,-rpath,/lib/intel64:/lib/intel64_win:/lib/win-x64:/opt/conda/envs/py_3.10/lib:/var/lib/jenkins/pytorch/build/lib:/opt/rocm-6.3.4/lib:/opt/rocm/lib:  lib/libgtest_main.a  -lstdc++  -Wl,--no-as-needed,"/var/lib/jenkins/pytorch/build/lib/libtorch.so" -Wl,--as-needed  -Wl,--no-as-needed,"/var/lib/jenkins/pytorch/build/lib/libtorch_cpu.so" -Wl,--as-needed  lib/libprotobuf.a  /opt/conda/envs/py_3.10/lib/libmkl_intel_lp64.so  /opt/conda/envs/py_3.10/lib/libmkl_gnu_thread.so  /opt/conda/envs/py_3.10/lib/libmkl_core.so  -fopenmp  /usr/lib/x86_64-linux-gnu/libpthread.a  -lm  /usr/lib/x86_64-linux-gnu/libdl.a  -Wl,--no-as-needed,"/var/lib/jenkins/pytorch/build/lib/libtorch_hip.so" -Wl,--as-needed  lib/libc10_hip.so  lib/libc10.so  /opt/rocm-6.3.4/lib/libMIOpen.so.1.0.60304  /opt/rocm/lib/libhiprtc.so.6.3.60304  -ldl  /opt/rocm-6.3.4/lib/libhipblas.so.2.3.60304  /opt/rocm-6.3.4/lib/libhipfft.so.0.1.60304  /opt/rocm-6.3.4/lib/libhiprand.so.1.1.60304  /opt/rocm-6.3.4/lib/librocrand.so.1.1.60304  /opt/rocm-6.3.4/lib/libhipsparse.so.1.1.0.60304  /opt/rocm-6.3.4/lib/libhipsolver.so.0.3.60304  /opt/rocm-6.3.4/lib/libhipblaslt.so.0.10.60304  /opt/rocm/lib/libamdhip64.so.6.3.60304  lib/libgtest.a  -Wl,-rpath-link,/opt/rocm-6.3.4/lib && /opt/conda/envs/py_3.10/bin/cmake -E __run_co_compile --lwyu="ldd;-u;-r" --source=bin/Dict_test && :
/usr/bin/ld: /var/lib/jenkins/pytorch/build/lib/libtorch_hip.so: undefined reference to `pytorch_flash::mem_eff_forward_ck(at::Tensor const&, at::Tensor const&, at::Tensor const&, float, bool, std::optional<bool>, std::optional<float>, std::optional<at::Tensor> const&, std::optional<at::Tensor>&, std::optional<at::Tensor> const&, std::optional<at::Tensor> const&, std::optional<at::Tensor> const&, std::optional<at::Tensor> const&, std::optional<at::Generator>, std::optional<at::Tensor>&)'
/usr/bin/ld: /var/lib/jenkins/pytorch/build/lib/libtorch_hip.so: undefined reference to `pytorch_flash::mha_varlen_bwd_ck(at::Tensor const&, at::Tensor const&, at::Tensor const&, at::Tensor const&, at::Tensor const&, at::Tensor const&, std::optional<at::Tensor>&, std::optional<at::Tensor>&, std::optional<at::Tensor>&, at::Tensor const&, at::Tensor const&, std::optional<at::Tensor>&, bool, std::optional<at::Tensor>&, int, int, float, float, bool, bool, int, int, bool, at::Tensor, at::Tensor)'
/usr/bin/ld: /var/lib/jenkins/pytorch/build/lib/libtorch_hip.so: undefined reference to `pytorch_flash::mem_eff_backward_ck(at::Tensor const&, at::Tensor const&, at::Tensor const&, at::Tensor const&, at::Tensor const&, at::Tensor const&, at::Tensor const&, at::Tensor const&, at::Tensor const&, std::optional<at::Tensor>&, bool, std::optional<at::Tensor>&, std::optional<at::Tensor>&, std::optional<at::Tensor>&, int, int, float, float, bool, bool, bool, at::Tensor, at::Tensor)'
/usr/bin/ld: /var/lib/jenkins/pytorch/build/lib/libtorch_hip.so: undefined reference to `pytorch_flash::mha_fwd_ck(at::Tensor const&, at::Tensor const&, at::Tensor const&, std::optional<at::Tensor>&, float, float, bool, int, int, bool, std::optional<at::Generator>, std::optional<at::Tensor> const&)'

rocm-repo-management-api · 2025-03-27T03:41:02Z

Jenkins build for cc09d84cab096e6aeca0d3d088b694746f363164 commit finished as FAILURE
Links: Blue Ocean view / Build artifacts

Detected error during Pytorch building:

	/var/lib/jenkins/pytorch/build/lib/libtorch_cpu.so
	/var/lib/jenkins/pytorch/build/lib/libtorch_hip.so
[7789/7886] Building CXX object caffe2/torch/CMakeFiles/torch_python.dir/csrc/distributed/rpc/init.cpp.o
cc1plus: warning: command-line option ‘-Wno-duplicate-decl-specifier’ is valid for C/ObjC but not for C++
[7790/7886] Linking CXX executable bin/static_runtime_bench
FAILED: bin/static_runtime_bench 
: && /opt/cache/bin/c++ -D_GLIBCXX_USE_CXX11_ABI=1 -fvisibility-inlines-hidden -DUSE_PTHREADPOOL -DNDEBUG -DUSE_KINETO -DLIBKINETO_NOCUPTI -DLIBKINETO_NOXPUPTI=ON -DUSE_FBGEMM -DUSE_PYTORCH_QNNPACK -DUSE_XNNPACK -DSYMBOLICATE_MOBILE_DEBUG_HANDLE -O2 -fPIC -Wall -Wextra -Werror=return-type -Werror=non-virtual-dtor -Werror=range-loop-construct -Werror=bool-operation -Wnarrowing -Wno-missing-field-initializers -Wno-type-limits -Wno-array-bounds -Wno-unknown-pragmas -Wno-unused-parameter -Wno-strict-overflow -Wno-strict-aliasing -Wno-stringop-overflow -Wsuggest-override -Wno-psabi -Wno-error=old-style-cast -Wno-missing-braces -fdiagnostics-color=always -faligned-new -Wno-unused-but-set-variable -Wno-maybe-uninitialized -fno-math-errno -fno-trapping-math -Werror=format -Wno-stringop-overflow -DHAVE_AVX512_CPU_DEFINITION -DHAVE_AVX2_CPU_DEFINITION -O3 -DNDEBUG -DNDEBUG -rdynamic     -Wl,--no-as-needed caffe2/CMakeFiles/static_runtime_bench.dir/__/benchmarks/static_runtime/deep_wide_pt.cc.o caffe2/CMakeFiles/static_runtime_bench.dir/__/benchmarks/static_runtime/deep_wide_pt_bench.cc.o -o bin/static_runtime_bench -L/lib/intel64   -L/lib/intel64_win   -L/lib/win-x64 -Wl,-rpath,/lib/intel64:/lib/intel64_win:/lib/win-x64:/opt/conda/envs/py_3.10/lib:/var/lib/jenkins/pytorch/build/lib:/opt/rocm-6.3.4/lib:/opt/rocm/lib  lib/libbenchmark.a  -Wl,--no-as-needed,"/var/lib/jenkins/pytorch/build/lib/libtorch.so" -Wl,--as-needed  -Wl,--no-as-needed,"/var/lib/jenkins/pytorch/build/lib/libtorch_cpu.so" -Wl,--as-needed  lib/libprotobuf.a  /opt/conda/envs/py_3.10/lib/libmkl_intel_lp64.so  /opt/conda/envs/py_3.10/lib/libmkl_gnu_thread.so  /opt/conda/envs/py_3.10/lib/libmkl_core.so  -fopenmp  /usr/lib/x86_64-linux-gnu/libpthread.a  -lm  /usr/lib/x86_64-linux-gnu/libdl.a  -Wl,--no-as-needed,"/var/lib/jenkins/pytorch/build/lib/libtorch_hip.so" -Wl,--as-needed  lib/libc10_hip.so  lib/libc10.so  /opt/rocm-6.3.4/lib/libMIOpen.so.1.0.60304  /opt/rocm/lib/libhiprtc.so.6.3.60304  -ldl  /opt/rocm-6.3.4/lib/libhipblas.so.2.3.60304  /opt/rocm-6.3.4/lib/libhipfft.so.0.1.60304  /opt/rocm-6.3.4/lib/libhiprand.so.1.1.60304  /opt/rocm-6.3.4/lib/librocrand.so.1.1.60304  /opt/rocm-6.3.4/lib/libhipsparse.so.1.1.0.60304  /opt/rocm-6.3.4/lib/libhipsolver.so.0.3.60304  /opt/rocm-6.3.4/lib/libhipblaslt.so.0.10.60304  /opt/rocm/lib/libamdhip64.so.6.3.60304  -lrt  -Wl,-rpath-link,/opt/rocm-6.3.4/lib && /opt/conda/envs/py_3.10/bin/cmake -E __run_co_compile --lwyu="ldd;-u;-r" --source=bin/static_runtime_bench && :
/usr/bin/ld: /var/lib/jenkins/pytorch/build/lib/libtorch_hip.so: undefined reference to `pytorch_flash::mem_eff_forward_ck(at::Tensor const&, at::Tensor const&, at::Tensor const&, float, bool, std::optional<bool>, std::optional<float>, std::optional<at::Tensor> const&, std::optional<at::Tensor>&, std::optional<at::Tensor> const&, std::optional<at::Tensor> const&, std::optional<at::Tensor> const&, std::optional<at::Tensor> const&, std::optional<at::Generator>, std::optional<at::Tensor>&)'
/usr/bin/ld: /var/lib/jenkins/pytorch/build/lib/libtorch_hip.so: undefined reference to `pytorch_flash::mha_varlen_bwd_ck(at::Tensor const&, at::Tensor const&, at::Tensor const&, at::Tensor const&, at::Tensor const&, at::Tensor const&, std::optional<at::Tensor>&, std::optional<at::Tensor>&, std::optional<at::Tensor>&, at::Tensor const&, at::Tensor const&, std::optional<at::Tensor>&, bool, std::optional<at::Tensor>&, int, int, float, float, bool, bool, int, int, bool, at::Tensor, at::Tensor)'
/usr/bin/ld: /var/lib/jenkins/pytorch/build/lib/libtorch_hip.so: undefined reference to `pytorch_flash::mem_eff_backward_ck(at::Tensor const&, at::Tensor const&, at::Tensor const&, at::Tensor const&, at::Tensor const&, at::Tensor const&, at::Tensor const&, at::Tensor const&, at::Tensor const&, std::optional<at::Tensor>&, bool, std::optional<at::Tensor>&, std::optional<at::Tensor>&, std::optional<at::Tensor>&, int, int, float, float, bool, bool, bool, at::Tensor, at::Tensor)'
/usr/bin/ld: /var/lib/jenkins/pytorch/build/lib/libtorch_hip.so: undefined reference to `pytorch_flash::mha_fwd_ck(at::Tensor const&, at::Tensor const&, at::Tensor const&, std::optional<at::Tensor>&, float, float, bool, int, int, bool, std::optional<at::Generator>, std::optional<at::Tensor> const&)'

CMakeLists.txt

benchmarks/transformer/sdpa.py

caffe2/CMakeLists.txt

CMakeLists.txt

cmake/External/ck.cmake

rocm-repo-management-api · 2025-03-27T20:44:01Z

Jenkins build for cc09d84cab096e6aeca0d3d088b694746f363164 commit finished as FAILURE
Links: Blue Ocean view / Build artifacts

Detected error during Pytorch building:

[7788/7886] Building CXX object caffe2/torch/CMakeFiles/torch_python.dir/csrc/utils/python_dispatch.cpp.o
cc1plus: warning: command-line option ‘-Wno-duplicate-decl-specifier’ is valid for C/ObjC but not for C++
[7789/7886] Building CXX object caffe2/torch/CMakeFiles/torch_python.dir/csrc/dynamo/guards.cpp.o
cc1plus: warning: command-line option ‘-Wno-duplicate-decl-specifier’ is valid for C/ObjC but not for C++
[7790/7886] Linking CXX executable bin/static_runtime_bench
FAILED: bin/static_runtime_bench 
: && /opt/cache/bin/c++ -D_GLIBCXX_USE_CXX11_ABI=1 -fvisibility-inlines-hidden -DUSE_PTHREADPOOL -DNDEBUG -DUSE_KINETO -DLIBKINETO_NOCUPTI -DLIBKINETO_NOXPUPTI=ON -DUSE_FBGEMM -DUSE_PYTORCH_QNNPACK -DUSE_XNNPACK -DSYMBOLICATE_MOBILE_DEBUG_HANDLE -O2 -fPIC -Wall -Wextra -Werror=return-type -Werror=non-virtual-dtor -Werror=range-loop-construct -Werror=bool-operation -Wnarrowing -Wno-missing-field-initializers -Wno-type-limits -Wno-array-bounds -Wno-unknown-pragmas -Wno-unused-parameter -Wno-strict-overflow -Wno-strict-aliasing -Wno-stringop-overflow -Wsuggest-override -Wno-psabi -Wno-error=old-style-cast -Wno-missing-braces -fdiagnostics-color=always -faligned-new -Wno-unused-but-set-variable -Wno-maybe-uninitialized -fno-math-errno -fno-trapping-math -Werror=format -Wno-stringop-overflow -DHAVE_AVX512_CPU_DEFINITION -DHAVE_AVX2_CPU_DEFINITION -O3 -DNDEBUG -DNDEBUG -rdynamic     -Wl,--no-as-needed caffe2/CMakeFiles/static_runtime_bench.dir/__/benchmarks/static_runtime/deep_wide_pt.cc.o caffe2/CMakeFiles/static_runtime_bench.dir/__/benchmarks/static_runtime/deep_wide_pt_bench.cc.o -o bin/static_runtime_bench -L/lib/intel64   -L/lib/intel64_win   -L/lib/win-x64 -Wl,-rpath,/lib/intel64:/lib/intel64_win:/lib/win-x64:/opt/conda/envs/py_3.10/lib:/var/lib/jenkins/pytorch/build/lib:/opt/rocm-6.3.4/lib:/opt/rocm/lib  lib/libbenchmark.a  -Wl,--no-as-needed,"/var/lib/jenkins/pytorch/build/lib/libtorch.so" -Wl,--as-needed  -Wl,--no-as-needed,"/var/lib/jenkins/pytorch/build/lib/libtorch_cpu.so" -Wl,--as-needed  lib/libprotobuf.a  /opt/conda/envs/py_3.10/lib/libmkl_intel_lp64.so  /opt/conda/envs/py_3.10/lib/libmkl_gnu_thread.so  /opt/conda/envs/py_3.10/lib/libmkl_core.so  -fopenmp  /usr/lib/x86_64-linux-gnu/libpthread.a  -lm  /usr/lib/x86_64-linux-gnu/libdl.a  -Wl,--no-as-needed,"/var/lib/jenkins/pytorch/build/lib/libtorch_hip.so" -Wl,--as-needed  lib/libc10_hip.so  lib/libc10.so  /opt/rocm-6.3.4/lib/libMIOpen.so.1.0.60304  /opt/rocm/lib/libhiprtc.so.6.3.60304  -ldl  /opt/rocm-6.3.4/lib/libhipblas.so.2.3.60304  /opt/rocm-6.3.4/lib/libhipfft.so.0.1.60304  /opt/rocm-6.3.4/lib/libhiprand.so.1.1.60304  /opt/rocm-6.3.4/lib/librocrand.so.1.1.60304  /opt/rocm-6.3.4/lib/libhipsparse.so.1.1.0.60304  /opt/rocm-6.3.4/lib/libhipsolver.so.0.3.60304  /opt/rocm-6.3.4/lib/libhipblaslt.so.0.10.60304  /opt/rocm/lib/libamdhip64.so.6.3.60304  -lrt  -Wl,-rpath-link,/opt/rocm-6.3.4/lib && /opt/conda/envs/py_3.10/bin/cmake -E __run_co_compile --lwyu="ldd;-u;-r" --source=bin/static_runtime_bench && :
/usr/bin/ld: /var/lib/jenkins/pytorch/build/lib/libtorch_hip.so: undefined reference to `pytorch_flash::mem_eff_forward_ck(at::Tensor const&, at::Tensor const&, at::Tensor const&, float, bool, std::optional<bool>, std::optional<float>, std::optional<at::Tensor> const&, std::optional<at::Tensor>&, std::optional<at::Tensor> const&, std::optional<at::Tensor> const&, std::optional<at::Tensor> const&, std::optional<at::Tensor> const&, std::optional<at::Generator>, std::optional<at::Tensor>&)'
/usr/bin/ld: /var/lib/jenkins/pytorch/build/lib/libtorch_hip.so: undefined reference to `pytorch_flash::mha_varlen_bwd_ck(at::Tensor const&, at::Tensor const&, at::Tensor const&, at::Tensor const&, at::Tensor const&, at::Tensor const&, std::optional<at::Tensor>&, std::optional<at::Tensor>&, std::optional<at::Tensor>&, at::Tensor const&, at::Tensor const&, std::optional<at::Tensor>&, bool, std::optional<at::Tensor>&, int, int, float, float, bool, bool, int, int, bool, at::Tensor, at::Tensor)'
/usr/bin/ld: /var/lib/jenkins/pytorch/build/lib/libtorch_hip.so: undefined reference to `pytorch_flash::mem_eff_backward_ck(at::Tensor const&, at::Tensor const&, at::Tensor const&, at::Tensor const&, at::Tensor const&, at::Tensor const&, at::Tensor const&, at::Tensor const&, at::Tensor const&, std::optional<at::Tensor>&, bool, std::optional<at::Tensor>&, std::optional<at::Tensor>&, std::optional<at::Tensor>&, int, int, float, float, bool, bool, bool, at::Tensor, at::Tensor)'
/usr/bin/ld: /var/lib/jenkins/pytorch/build/lib/libtorch_hip.so: undefined reference to `pytorch_flash::mha_fwd_ck(at::Tensor const&, at::Tensor const&, at::Tensor const&, std::optional<at::Tensor>&, float, float, bool, int, int, bool, std::optional<at::Generator>, std::optional<at::Tensor> const&)'

rocm-repo-management-api · 2025-04-01T18:04:18Z

Jenkins build for cc09d84cab096e6aeca0d3d088b694746f363164 commit finished as FAILURE
Links: Blue Ocean view / Build artifacts

Detected error during Pytorch building:

[7785/7886] Building CXX object caffe2/torch/CMakeFiles/torch_python.dir/csrc/profiler/python/init.cpp.o
cc1plus: warning: command-line option ‘-Wno-duplicate-decl-specifier’ is valid for C/ObjC but not for C++
[7786/7886] Building CXX object caffe2/torch/CMakeFiles/torch_python.dir/csrc/cuda/Module.cpp.o
cc1plus: warning: command-line option ‘-Wno-duplicate-decl-specifier’ is valid for C/ObjC but not for C++
[7787/7886] Linking CXX executable bin/static_runtime_bench
FAILED: bin/static_runtime_bench 
: && /opt/cache/bin/c++ -D_GLIBCXX_USE_CXX11_ABI=1 -fvisibility-inlines-hidden -DUSE_PTHREADPOOL -DNDEBUG -DUSE_KINETO -DLIBKINETO_NOCUPTI -DLIBKINETO_NOXPUPTI=ON -DUSE_FBGEMM -DUSE_PYTORCH_QNNPACK -DUSE_XNNPACK -DSYMBOLICATE_MOBILE_DEBUG_HANDLE -O2 -fPIC -Wall -Wextra -Werror=return-type -Werror=non-virtual-dtor -Werror=range-loop-construct -Werror=bool-operation -Wnarrowing -Wno-missing-field-initializers -Wno-type-limits -Wno-array-bounds -Wno-unknown-pragmas -Wno-unused-parameter -Wno-strict-overflow -Wno-strict-aliasing -Wno-stringop-overflow -Wsuggest-override -Wno-psabi -Wno-error=old-style-cast -Wno-missing-braces -fdiagnostics-color=always -faligned-new -Wno-unused-but-set-variable -Wno-maybe-uninitialized -fno-math-errno -fno-trapping-math -Werror=format -Wno-stringop-overflow -DHAVE_AVX512_CPU_DEFINITION -DHAVE_AVX2_CPU_DEFINITION -O3 -DNDEBUG -DNDEBUG -rdynamic     -Wl,--no-as-needed caffe2/CMakeFiles/static_runtime_bench.dir/__/benchmarks/static_runtime/deep_wide_pt.cc.o caffe2/CMakeFiles/static_runtime_bench.dir/__/benchmarks/static_runtime/deep_wide_pt_bench.cc.o -o bin/static_runtime_bench -L/lib/intel64   -L/lib/intel64_win   -L/lib/win-x64 -Wl,-rpath,/lib/intel64:/lib/intel64_win:/lib/win-x64:/opt/conda/envs/py_3.10/lib:/var/lib/jenkins/pytorch/build/lib:/opt/rocm-6.3.4/lib:/opt/rocm/lib  lib/libbenchmark.a  -Wl,--no-as-needed,"/var/lib/jenkins/pytorch/build/lib/libtorch.so" -Wl,--as-needed  -Wl,--no-as-needed,"/var/lib/jenkins/pytorch/build/lib/libtorch_cpu.so" -Wl,--as-needed  lib/libprotobuf.a  /opt/conda/envs/py_3.10/lib/libmkl_intel_lp64.so  /opt/conda/envs/py_3.10/lib/libmkl_gnu_thread.so  /opt/conda/envs/py_3.10/lib/libmkl_core.so  -fopenmp  /usr/lib/x86_64-linux-gnu/libpthread.a  -lm  /usr/lib/x86_64-linux-gnu/libdl.a  -Wl,--no-as-needed,"/var/lib/jenkins/pytorch/build/lib/libtorch_hip.so" -Wl,--as-needed  lib/libc10_hip.so  lib/libc10.so  /opt/rocm-6.3.4/lib/libMIOpen.so.1.0.60304  /opt/rocm/lib/libhiprtc.so.6.3.60304  -ldl  /opt/rocm-6.3.4/lib/libhipblas.so.2.3.60304  /opt/rocm-6.3.4/lib/libhipfft.so.0.1.60304  /opt/rocm-6.3.4/lib/libhiprand.so.1.1.60304  /opt/rocm-6.3.4/lib/librocrand.so.1.1.60304  /opt/rocm-6.3.4/lib/libhipsparse.so.1.1.0.60304  /opt/rocm-6.3.4/lib/libhipsolver.so.0.3.60304  /opt/rocm-6.3.4/lib/libhipblaslt.so.0.10.60304  /opt/rocm/lib/libamdhip64.so.6.3.60304  -lrt  -Wl,-rpath-link,/opt/rocm-6.3.4/lib && /opt/conda/envs/py_3.10/bin/cmake -E __run_co_compile --lwyu="ldd;-u;-r" --source=bin/static_runtime_bench && :
/usr/bin/ld: /var/lib/jenkins/pytorch/build/lib/libtorch_hip.so: undefined reference to `pytorch_flash::mem_eff_forward_ck(at::Tensor const&, at::Tensor const&, at::Tensor const&, float, bool, std::optional<bool>, std::optional<float>, std::optional<at::Tensor> const&, std::optional<at::Tensor>&, std::optional<at::Tensor> const&, std::optional<at::Tensor> const&, std::optional<at::Tensor> const&, std::optional<at::Tensor> const&, std::optional<at::Generator>, std::optional<at::Tensor>&)'
/usr/bin/ld: /var/lib/jenkins/pytorch/build/lib/libtorch_hip.so: undefined reference to `pytorch_flash::mha_varlen_bwd_ck(at::Tensor const&, at::Tensor const&, at::Tensor const&, at::Tensor const&, at::Tensor const&, at::Tensor const&, std::optional<at::Tensor>&, std::optional<at::Tensor>&, std::optional<at::Tensor>&, at::Tensor const&, at::Tensor const&, std::optional<at::Tensor>&, bool, std::optional<at::Tensor>&, int, int, float, float, bool, bool, int, int, bool, at::Tensor, at::Tensor)'
/usr/bin/ld: /var/lib/jenkins/pytorch/build/lib/libtorch_hip.so: undefined reference to `pytorch_flash::mem_eff_backward_ck(at::Tensor const&, at::Tensor const&, at::Tensor const&, at::Tensor const&, at::Tensor const&, at::Tensor const&, at::Tensor const&, at::Tensor const&, at::Tensor const&, std::optional<at::Tensor>&, bool, std::optional<at::Tensor>&, std::optional<at::Tensor>&, std::optional<at::Tensor>&, int, int, float, float, bool, bool, bool, at::Tensor, at::Tensor)'
/usr/bin/ld: /var/lib/jenkins/pytorch/build/lib/libtorch_hip.so: undefined reference to `pytorch_flash::mha_fwd_ck(at::Tensor const&, at::Tensor const&, at::Tensor const&, std::optional<at::Tensor>&, float, float, bool, int, int, bool, std::optional<at::Generator>, std::optional<at::Tensor> const&)'

rocm-repo-management-api · 2025-04-02T04:05:05Z

Jenkins build for cc09d84cab096e6aeca0d3d088b694746f363164 commit finished as FAILURE
Links: Blue Ocean view / Build artifacts

Detected error during Pytorch building:

      |   ~~~~^~~~~~~~~~
/var/lib/jenkins/pytorch/c10/util/BFloat16.h:33:12: note: ‘tmp’ declared here
   33 |   uint32_t tmp = src;
      |            ^~~
[7377/7886] Building CXX object caffe2/CMakeFiles/torch_hip.dir/__/aten/src/ATen/native/hip/Blas.cpp.o
FAILED: caffe2/CMakeFiles/torch_hip.dir/__/aten/src/ATen/native/hip/Blas.cpp.o 
/opt/cache/bin/sccache /opt/cache/bin/c++ -DAT_PER_OPERATOR_HEADERS -DFLASHATTENTION_DISABLE_ALIBI -DFMT_HEADER_ONLY=1 -DHAVE_MALLOC_USABLE_SIZE=1 -DHAVE_MMAP=1 -DHAVE_SHM_OPEN=1 -DHAVE_SHM_UNLINK=1 -DIDEEP_USE_MKL -DMINIZ_DISABLE_ZIP_READER_CRC32_CHECKS -DONNXIFI_ENABLE_EXT=1 -DONNX_ML=1 -DONNX_NAMESPACE=onnx_torch -DPYTORCH_LAYERNORM_FAST_RECIPROCAL -DROCM_VERSION=60304 -DTORCH_ENABLE_LLVM -DTORCH_HIP_BUILD_MAIN_LIB -DTORCH_HIP_VERSION=603 -DUSE_C10D_GLOO -DUSE_C10D_NCCL -DUSE_CK_FLASH_ATTENTION -DUSE_DISTRIBUTED -DUSE_EXTERNAL_MZCRC -DUSE_FLASH_ATTENTION -DUSE_MEM_EFF_ATTENTION -DUSE_NCCL -DUSE_PROF_API=1 -DUSE_ROCM -DUSE_RPC -DUSE_TENSORPIPE -D_FILE_OFFSET_BITS=64 -D__HIP_PLATFORM_AMD__ -D__HIP_PLATFORM_AMD__=1 -Dtorch_hip_EXPORTS -I/var/lib/jenkins/pytorch/build/aten/src -I/var/lib/jenkins/pytorch/aten/src -I/var/lib/jenkins/pytorch/build -I/var/lib/jenkins/pytorch -I/var/lib/jenkins/pytorch/cmake/../third_party/benchmark/include -I/opt/llvm/include -I/var/lib/jenkins/pytorch/third_party/onnx -I/var/lib/jenkins/pytorch/build/third_party/onnx -I/var/lib/jenkins/pytorch/nlohmann -I/opt/rocm/hcc/include -I/opt/rocm/rocblas/include -I/opt/rocm/hipsparse/include -I/opt/rocm/include/rccl -I/var/lib/jenkins/pytorch/aten/src/THH -I/var/lib/jenkins/pytorch/aten/src/ATen/hip -I/var/lib/jenkins/pytorch/aten/src/ATen/../../../third_party/composable_kernel/include -I/var/lib/jenkins/pytorch/aten/src/ATen/../../../third_party/composable_kernel/library/include -I/var/lib/jenkins/pytorch/third_party/fmt/include -I/var/lib/jenkins/pytorch/aten/src/ATen/native/transformers/hip/flash_attn/ck -I/var/lib/jenkins/pytorch/build/caffe2/aten/src -I/var/lib/jenkins/pytorch/aten/src/ATen/.. -I/var/lib/jenkins/pytorch/torch/include -I/var/lib/jenkins/pytorch/c10/hip/../.. -I/var/lib/jenkins/pytorch/c10/.. -I/var/lib/jenkins/pytorch/torch/csrc/api -I/var/lib/jenkins/pytorch/torch/csrc/api/include -I/var/lib/jenkins/pytorch/build/third_party/gloo/hip -isystem /opt/rocm-6.3.4/include -isystem /var/lib/jenkins/pytorch/build/third_party/gloo -isystem /var/lib/jenkins/pytorch/cmake/../third_party/gloo -isystem /var/lib/jenkins/pytorch/cmake/../third_party/tensorpipe/third_party/libuv/include -isystem /var/lib/jenkins/pytorch/cmake/../third_party/googletest/googlemock/include -isystem /var/lib/jenkins/pytorch/cmake/../third_party/googletest/googletest/include -isystem /var/lib/jenkins/pytorch/third_party/protobuf/src -isystem /opt/conda/envs/py_3.10/include -isystem /var/lib/jenkins/pytorch/third_party/XNNPACK/include -isystem /var/lib/jenkins/pytorch/third_party/ittapi/include -isystem /var/lib/jenkins/pytorch/cmake/../third_party/eigen -isystem /var/lib/jenkins/pytorch/third_party/ideep/mkl-dnn/include/oneapi/dnnl -isystem /var/lib/jenkins/pytorch/third_party/ideep/include -isystem /var/lib/jenkins/pytorch/INTERFACE -isystem /var/lib/jenkins/pytorch/third_party/nlohmann/include -isystem /opt/rocm/include -isystem /opt/rocm-6.3.4/include/hiprand -isystem /opt/rocm-6.3.4/include/rocrand -isystem /opt/rocm/magma/include -D_GLIBCXX_USE_CXX11_ABI=1 -fvisibility-inlines-hidden -DUSE_PTHREADPOOL -DNDEBUG -DUSE_KINETO -DLIBKINETO_NOCUPTI -DLIBKINETO_NOXPUPTI=ON -DUSE_FBGEMM -DUSE_PYTORCH_QNNPACK -DUSE_XNNPACK -DSYMBOLICATE_MOBILE_DEBUG_HANDLE -O2 -fPIC -Wall -Wextra -Werror=return-type -Werror=non-virtual-dtor -Werror=range-loop-construct -Werror=bool-operation -Wnarrowing -Wno-missing-field-initializers -Wno-type-limits -Wno-array-bounds -Wno-unknown-pragmas -Wno-unused-parameter -Wno-strict-overflow -Wno-strict-aliasing -Wno-stringop-overflow -Wsuggest-override -Wno-psabi -Wno-error=old-style-cast -Wno-missing-braces -fdiagnostics-color=always -faligned-new -Wno-unused-but-set-variable -Wno-maybe-uninitialized -fno-math-errno -fno-trapping-math -Werror=format -Wno-stringop-overflow -DHAVE_AVX512_CPU_DEFINITION -DHAVE_AVX2_CPU_DEFINITION -O3 -DNDEBUG -DNDEBUG -std=gnu++17 -fPIC -DMKL_HAS_SBGEMM -DTORCH_USE_LIBUV -DCAFFE2_USE_GLOO -Wall -Wextra -Wdeprecated -Wno-unused-parameter -Wno-missing-field-initializers -Wno-array-bounds -Wno-unknown-pragmas -Wno-strict-overflow -Wno-strict-aliasing -Wunused-function -Wunused-variable -Wunused-but-set-variable -Wno-maybe-uninitialized -fvisibility=hidden -O2 -fPIC -D__HIP_PLATFORM_AMD__=1 -DCUDA_HAS_FP16=1 -DUSE_ROCM -D__HIP_NO_HALF_OPERATORS__=1 -D__HIP_NO_HALF_CONVERSIONS__=1 -DTORCH_HIP_VERSION=603 -Wno-shift-count-negative -Wno-shift-count-overflow -Wno-duplicate-decl-specifier -DCAFFE2_USE_MIOPEN -DTHRUST_DEVICE_SYSTEM=THRUST_DEVICE_SYSTEM_HIP -std=c++17 -DHIPBLAS_V2 -DHIPBLASLT_VEC_EXT -D_GLIBCXX_USE_CXX11_ABI=1 -DHIP_ENABLE_WARP_SYNC_BUILTINS -DHIP_VERSION=6 -DUSE_MIOPEN -MD -MT caffe2/CMakeFiles/torch_hip.dir/__/aten/src/ATen/native/hip/Blas.cpp.o -MF caffe2/CMakeFiles/torch_hip.dir/__/aten/src/ATen/native/hip/Blas.cpp.o.d -o caffe2/CMakeFiles/torch_hip.dir/__/aten/src/ATen/native/hip/Blas.cpp.o -c /var/lib/jenkins/pytorch/aten/src/ATen/native/hip/Blas.cpp
cc1plus: warning: command-line option ‘-Wno-duplicate-decl-specifier’ is valid for C/ObjC but not for C++
In file included from /var/lib/jenkins/pytorch/aten/src/ATen/native/hip/Blas.cpp:15:
/var/lib/jenkins/pytorch/aten/src/ATen/hip/tunable/TunableGemm.h:25:10: fatal error: c10/util/Float8_e8m0fnu.h: No such file or directory
   25 | #include <c10/util/Float8_e8m0fnu.h>

akashveramd · 2025-04-03T02:27:26Z

@pruthvistony @jithunnair-amd @alugorey
Since I mistakenly created release/2.6_ck branch as a protected branch. Hence I cannot push my new commits added to address review comments. Therefore, I have created a new branch and a new PR which addresses the review comments. Here is the link to the new PR-
#2016

rocm-repo-management-api · 2025-04-03T03:09:23Z

Jenkins build for cc09d84cab096e6aeca0d3d088b694746f363164 commit is in progress
Links: Blue Ocean view / Build artifacts

jithunnair-amd · 2025-04-03T11:51:52Z

Closing in favor of #2016

Added the code to download CK library from compute-artifactory and cr…

cc09d84

…eate link target. Enable USE_CK_FLASH_ATTENTION based on USE_FLASH_ATTENTION option.

akashveramd requested review from pragupta, pruthvistony, jithunnair-amd and alugorey March 26, 2025 22:56

akashveramd self-assigned this Mar 26, 2025

pruthvistony requested changes Mar 27, 2025

View reviewed changes

CMakeLists.txt Show resolved Hide resolved

benchmarks/transformer/sdpa.py Show resolved Hide resolved

caffe2/CMakeLists.txt Show resolved Hide resolved

CMakeLists.txt Show resolved Hide resolved

cmake/External/ck.cmake Show resolved Hide resolved

akashveramd mentioned this pull request Apr 1, 2025

Merge release/2.6_ck_2 to release/2.6 #2012

Closed

akashveramd mentioned this pull request Apr 3, 2025

Merge CK validation to release/2.6 #2016

Closed

jithunnair-amd closed this Apr 3, 2025

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Download CK library from compute-artifactory and link to Pytorch #2007

Download CK library from compute-artifactory and link to Pytorch #2007

akashveramd commented Mar 26, 2025

rocm-repo-management-api bot commented Mar 26, 2025 •

edited

Loading

rocm-repo-management-api bot commented Mar 27, 2025 •

edited

Loading

rocm-repo-management-api bot commented Mar 27, 2025 •

edited

Loading

rocm-repo-management-api bot commented Apr 1, 2025 •

edited

Loading

rocm-repo-management-api bot commented Apr 2, 2025 •

edited

Loading

akashveramd commented Apr 3, 2025 •

edited

Loading

rocm-repo-management-api bot commented Apr 3, 2025

jithunnair-amd commented Apr 3, 2025

Download CK library from compute-artifactory and link to Pytorch #2007

Download CK library from compute-artifactory and link to Pytorch #2007

Conversation

akashveramd commented Mar 26, 2025

rocm-repo-management-api bot commented Mar 26, 2025 • edited Loading

rocm-repo-management-api bot commented Mar 27, 2025 • edited Loading

rocm-repo-management-api bot commented Mar 27, 2025 • edited Loading

rocm-repo-management-api bot commented Apr 1, 2025 • edited Loading

rocm-repo-management-api bot commented Apr 2, 2025 • edited Loading

akashveramd commented Apr 3, 2025 • edited Loading

rocm-repo-management-api bot commented Apr 3, 2025

jithunnair-amd commented Apr 3, 2025

rocm-repo-management-api bot commented Mar 26, 2025 •

edited

Loading

rocm-repo-management-api bot commented Mar 27, 2025 •

edited

Loading

rocm-repo-management-api bot commented Mar 27, 2025 •

edited

Loading

rocm-repo-management-api bot commented Apr 1, 2025 •

edited

Loading

rocm-repo-management-api bot commented Apr 2, 2025 •

edited

Loading

akashveramd commented Apr 3, 2025 •

edited

Loading