[BUG]LIBRAFT cannot be built when RMM locally installed

Question

[BUG]LIBRAFT cannot be built when RMM locally installed

gdaisukesuzuki opened this issue 5 months ago · comments

Describe the bug
When I tried to build libraft in an environment where rmm is already installed, I encountered an error stating that 'CCCL::Thrust' is already registered and thus cannot be created. I would appreciate it if you could advise me on how to circumvent this issue or provide a corresponding patch.

The environment is as follows:

BRANCH=branch-24.02
bundled
CUDA 12.3.1 (CCCL 2.2.0 bundled)

Steps/Code to reproduce bug

$ ./build.sh -n -v libraft
grep: warning: stray \ before -
grep: warning: stray \ before -
grep: warning: stray \ before -
grep: warning: stray \ before -
grep: warning: stray \ before -
grep: warning: stray \ before -
grep: warning: stray \ before -
grep: warning: stray \ before -
grep: warning: stray \ before -
grep: warning: stray \ before -
grep: warning: stray \ before -
grep: warning: stray \ before -
grep: warning: stray \ before -
grep: warning: stray \ before -
grep: warning: stray \ before -
grep: warning: stray \ before -
grep: warning: stray \ before -
grep: warning: stray \ before -
Building for the architecture of the GPU in the system...
-- The CXX compiler identification is GNU 13.2.1
-- The CUDA compiler identification is NVIDIA 12.3.103
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Check for working CXX compiler: /usr/bin/c++ - skipped
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Detecting CUDA compiler ABI info
-- Detecting CUDA compiler ABI info - done
-- Check for working CUDA compiler: /opt/cuda/bin/nvcc - skipped
-- Detecting CUDA compile features
-- Detecting CUDA compile features - done
-- Auto detection of gpu-archs: 89
-- RAFT: Building optional components: 
-- RAFT: Build RAFT unit-tests: OFF
-- RAFT: Building raft C++ benchmarks: OFF
-- RAFT: Building ANN benchmarks: OFF
-- RAFT: Build CPU only components: OFF
-- RAFT: Enable detection of conda environment for dependencies: ON
-- RAFT: Disable depreaction warnings ON
-- RAFT: Disable OpenMP: OFF
-- RAFT: Enable kernel resource usage info: OFF
-- RAFT: Enable lineinfo in nvcc: OFF
-- RAFT: Enable nvtx markers: ON
-- RAFT: Statically link the CUDA toolkit runtime and libraries: OFF
-- RAFT: RMM_LOGGING_LEVEL = 'INFO'.
-- Found CUDAToolkit: /opt/cuda/targets/x86_64-linux/include (found version "12.3.103") 
-- Performing Test CMAKE_HAVE_LIBC_PTHREAD
-- Performing Test CMAKE_HAVE_LIBC_PTHREAD - Success
-- Found Threads: TRUE  
-- Found OpenMP_CXX: -fopenmp (found version "4.5") 
-- Found OpenMP: TRUE (found version "4.5")  
-- RAFT: OpenMP found in 
-- Downloading CPM.cmake to /home/dai/work/raft/raft/cpp/build/cmake/CPM_0.38.5.cmake
-- Found Git: /usr/bin/git (found version "2.43.0") 
-- Thrust: TargetInfo: Thrust::Thrust: (2.2.0.0)
-- Thrust: TargetInfo: Thrust::Thrust > ALIASED_TARGET: _Thrust_Thrust
-- Thrust: TargetInfo: Thrust::Thrust > INTERFACE_INCLUDE_DIRECTORIES: /opt/cuda/include
-- Thrust: Searching for libcudacxx REQUIRED
-- Thrust: Setting libcudacxx target to libcudacxx::libcudacxx
-- Thrust: TargetInfo: libcudacxx::libcudacxx: (2.2.0.0)
-- Thrust: TargetInfo: libcudacxx::libcudacxx > IMPORTED: TRUE
-- Thrust: TargetInfo: libcudacxx::libcudacxx > INTERFACE_LINK_LIBRARIES: _libcudacxx_libcudacxx
-- Thrust: TargetInfo: Thrust::libcudacxx: (libcudacxx 2.2.0.0)
-- Thrust: TargetInfo: Thrust::libcudacxx > ALIASED_TARGET: _Thrust_libcudacxx
-- Thrust: TargetInfo: Thrust::libcudacxx > INTERFACE_LINK_LIBRARIES: libcudacxx::libcudacxx
-- Thrust: Assembling target CCCL::Thrust. Options: FROM_OPTIONS;HOST_OPTION;CCCL_THRUST_HOST_SYSTEM;DEVICE_OPTION;CCCL_THRUST_DEVICE_SYSTEM;HOST_OPTION_DOC;Host system for CCCL::Thrust target.;DEVICE_OPTION_DOC;Device system for CCCL::Thrust target.;ADVANCED
-- Thrust: Creating system cache options: (advanced=TRUE)
-- Thrust:   - Host Option=CCCL_THRUST_HOST_SYSTEM Default=CPP Doc='Host system for CCCL::Thrust target.'
-- Thrust:   - Device Option=CCCL_THRUST_DEVICE_SYSTEM Default=CUDA Doc='Device system for CCCL::Thrust target.'
-- Thrust: Current option settings:
-- Thrust:   - CCCL_THRUST_HOST_SYSTEM=CPP
-- Thrust:   - CCCL_THRUST_DEVICE_SYSTEM=CUDA
-- Thrust: Generating CPP targets.
-- Thrust: TargetInfo: Thrust::CPP: (Thrust 2.2.0.0)
-- Thrust: TargetInfo: Thrust::CPP > ALIASED_TARGET: _Thrust_CPP
-- Thrust: TargetInfo: Thrust::CPP > INTERFACE_LINK_LIBRARIES: Thrust::Thrust
-- Thrust: TargetInfo: Thrust::CPP::Host: 
-- Thrust: TargetInfo: Thrust::CPP::Host > ALIASED_TARGET: _Thrust_CPP_Host
-- Thrust: TargetInfo: Thrust::CPP::Host > INTERFACE_COMPILE_DEFINITIONS: THRUST_HOST_SYSTEM=THRUST_HOST_SYSTEM_CPP
-- Thrust: TargetInfo: Thrust::CPP::Host > INTERFACE_LINK_LIBRARIES: Thrust::CPP
-- Thrust: TargetInfo: Thrust::CPP::Host > INTERFACE_THRUST_HOST: CPP
-- Thrust: TargetInfo: Thrust::CPP::Device: 
-- Thrust: TargetInfo: Thrust::CPP::Device > ALIASED_TARGET: _Thrust_CPP_Device
-- Thrust: TargetInfo: Thrust::CPP::Device > INTERFACE_COMPILE_DEFINITIONS: THRUST_DEVICE_SYSTEM=THRUST_DEVICE_SYSTEM_CPP
-- Thrust: TargetInfo: Thrust::CPP::Device > INTERFACE_LINK_LIBRARIES: Thrust::CPP
-- Thrust: TargetInfo: Thrust::CPP::Device > INTERFACE_THRUST_DEVICE: CPP
-- Thrust: Searching for CUB REQUIRED
-- Thrust: Setting CUB target to CUB::CUB
-- Thrust: TargetInfo: CUB::CUB: (2.2.0.0)
-- Thrust: TargetInfo: CUB::CUB > ALIASED_TARGET: _CUB_CUB
-- Thrust: TargetInfo: CUB::CUB > INTERFACE_INCLUDE_DIRECTORIES: /opt/cuda/include
-- Thrust: TargetInfo: CUB::CUB > INTERFACE_LINK_LIBRARIES: CUB::libcudacxx
-- Thrust: TargetInfo: Thrust::CUB: (CUB 2.2.0.0)
-- Thrust: TargetInfo: Thrust::CUB > ALIASED_TARGET: _Thrust_CUB
-- Thrust: TargetInfo: Thrust::CUB > INTERFACE_LINK_LIBRARIES: CUB::CUB
-- Thrust: TargetInfo: Thrust::CUDA::Device: 
-- Thrust: TargetInfo: Thrust::CUDA::Device > ALIASED_TARGET: _Thrust_CUDA_Device
-- Thrust: TargetInfo: Thrust::CUDA::Device > INTERFACE_COMPILE_DEFINITIONS: THRUST_DEVICE_SYSTEM=THRUST_DEVICE_SYSTEM_CUDA
-- Thrust: TargetInfo: Thrust::CUDA::Device > INTERFACE_LINK_LIBRARIES: Thrust::CUDA
-- Thrust: TargetInfo: Thrust::CUDA::Device > INTERFACE_THRUST_DEVICE: CUDA
-- Thrust: TargetInfo: Thrust::CUDA: 
-- Thrust: TargetInfo: Thrust::CUDA > ALIASED_TARGET: _Thrust_CUDA
-- Thrust: TargetInfo: Thrust::CUDA > INTERFACE_LINK_LIBRARIES: Thrust::Thrust;Thrust::CUB
-- Thrust: TargetInfo: CCCL::Thrust: (Thrust 2.2.0.0)
-- Thrust: TargetInfo: CCCL::Thrust > IMPORTED: TRUE
-- Thrust: TargetInfo: CCCL::Thrust > INTERFACE_LINK_LIBRARIES: Thrust::CPP::Host;Thrust::CUDA::Device
-- CPM: Using local package CCCL@2.2.0.0
-- Thrust: Assembling target CCCL::Thrust. Options: FROM_OPTIONS;HOST_OPTION;CCCL_THRUST_HOST_SYSTEM;DEVICE_OPTION;CCCL_THRUST_DEVICE_SYSTEM;HOST_OPTION_DOC;Host system for CCCL::Thrust target.;DEVICE_OPTION_DOC;Device system for CCCL::Thrust target.;ADVANCED
-- Thrust: Creating system cache options: (advanced=TRUE)
-- Thrust:   - Host Option=CCCL_THRUST_HOST_SYSTEM Default=CPP Doc='Host system for CCCL::Thrust target.'
-- Thrust:   - Device Option=CCCL_THRUST_DEVICE_SYSTEM Default=CUDA Doc='Device system for CCCL::Thrust target.'
-- Thrust: Current option settings:
-- Thrust:   - CCCL_THRUST_HOST_SYSTEM=CPP
-- Thrust:   - CCCL_THRUST_DEVICE_SYSTEM=CUDA
CMake Error at /opt/cuda/lib/cmake/thrust/thrust-config.cmake:183 (add_library):
  add_library cannot create imported target "CCCL::Thrust" because another
  target with the same name already exists.
Call Stack (most recent call first):
  /opt/cuda/lib/cmake/cccl/cccl-config.cmake:82 (thrust_create_target)
  /usr/lib/cmake/rmm/rmm-dependencies.cmake:27 (find_package)
  /usr/lib/cmake/rmm/rmm-config.cmake:84 (include)
  build/cmake/CPM_0.38.5.cmake:243 (find_package)
  build/cmake/CPM_0.38.5.cmake:303 (cpm_find_package)
  build/_deps/rapids-cmake-src/rapids-cmake/cpm/find.cmake:176 (CPMFindPackage)
  build/_deps/rapids-cmake-src/rapids-cmake/cpm/rmm.cmake:75 (rapids_cpm_find)
  cmake/thirdparty/get_rmm.cmake:19 (rapids_cpm_rmm)
  cmake/thirdparty/get_rmm.cmake:23 (find_and_configure_rmm)
  CMakeLists.txt:182 (include)


-- Thrust: Assembling target CCCL::Thrust. Options: FROM_OPTIONS;HOST_OPTION;CCCL_THRUST_HOST_SYSTEM;DEVICE_OPTION;CCCL_THRUST_DEVICE_SYSTEM;HOST_OPTION_DOC;Host system for CCCL::Thrust target.;DEVICE_OPTION_DOC;Device system for CCCL::Thrust target.;ADVANCED
-- Thrust: Creating system cache options: (advanced=TRUE)
-- Thrust:   - Host Option=CCCL_THRUST_HOST_SYSTEM Default=CPP Doc='Host system for CCCL::Thrust target.'
-- Thrust:   - Device Option=CCCL_THRUST_DEVICE_SYSTEM Default=CUDA Doc='Device system for CCCL::Thrust target.'
-- Thrust: Current option settings:
-- Thrust:   - CCCL_THRUST_HOST_SYSTEM=CPP
-- Thrust:   - CCCL_THRUST_DEVICE_SYSTEM=CUDA
CMake Error at /opt/cuda/lib/cmake/thrust/thrust-config.cmake:183 (add_library):
  add_library cannot create imported target "CCCL::Thrust" because another
  target with the same name already exists.
Call Stack (most recent call first):
  /opt/cuda/lib/cmake/cccl/cccl-config.cmake:82 (thrust_create_target)
  /usr/share/cmake/Modules/CMakeFindDependencyMacro.cmake:76 (find_package)
  /usr/lib/cmake/rmm/rmm-dependencies.cmake:28 (find_dependency)
  /usr/lib/cmake/rmm/rmm-config.cmake:84 (include)
  build/cmake/CPM_0.38.5.cmake:243 (find_package)
  build/cmake/CPM_0.38.5.cmake:303 (cpm_find_package)
  build/_deps/rapids-cmake-src/rapids-cmake/cpm/find.cmake:176 (CPMFindPackage)
  build/_deps/rapids-cmake-src/rapids-cmake/cpm/rmm.cmake:75 (rapids_cpm_find)
  cmake/thirdparty/get_rmm.cmake:19 (rapids_cpm_rmm)
  cmake/thirdparty/get_rmm.cmake:23 (find_and_configure_rmm)
  CMakeLists.txt:182 (include)


-- CPM: Using local package rmm@24.02.0
-- CPM: Adding package NvidiaCutlass@2.10.0 (v2.10.0)
-- CMake Version: 3.28.1
-- CUDART: /opt/cuda/lib64/libcudart.so
-- CUDA Driver: /opt/cuda/lib64/stubs/libcuda.so
-- NVRTC: /opt/cuda/lib64/libnvrtc.so
-- Default Install Location: /home/dai/work/raft/raft/cpp/build/install
-- CUDA Compilation Architectures: 53;60;61;70;72;75;80;86
-- Enable caching of reference results in conv unit tests
-- Enable rigorous conv problem sizes in conv unit tests
-- Using NVCC flags: -DCUTLASS_TEST_LEVEL=0;-DCUTLASS_TEST_ENABLE_CACHED_RESULTS=1;-DCUTLASS_CONV_UNIT_TEST_RIGOROUS_SIZE_ENABLED=1;-DCUTLASS_DEBUG_TRACE_LEVEL=0;$<$<BOOL:1>:-Xcompiler=-Wconversion>;$<$<BOOL:1>:-Xcompiler=-fno-strict-aliasing>
-- CUTLASS Revision: bae049bf
-- Configuring cublas ...
-- cuBLAS Disabled.
-- Configuring cuBLAS ... done.
-- Configuring cuDNN ...
-- cuDNN: /lib64/libcudnn.so
-- cuDNN: /usr/include
-- Configuring cuDNN ... done.
-- CPM: Adding package cuco@0.0.1 (f823d30d6b08a60383266db25821074dbdbe5822)
-- Configuring incomplete, errors occurred!

Environment details (please complete the following information):

Environment location: [local]
Method of RAFT install: [build from source]

Robert Maynard · Answer 1 · Tue Jan 09 2024 01:07:57 GMT+0800 (China Standard Time)

You have two options:

Patch the cccl-config.cmake provided by the CUDA Toolkit as it has a bug ( fix is: https://github.com/NVIDIA/cccl/pull/1157/files#diff-3d1a983827676facf0f75944c8eae95c4b55e01b7bb98b1317d1ef90ce2d17e6L74 )
Use the CCCL provided by rmm as it has the above fix already applied, and otherwise is the same

gdaisukesuzuki · Answer 2 · Fri Jan 12 2024 14:00:18 GMT+0800 (China Standard Time)

@robertmaynard Thank you for the comment. I was able to resolve the issue by applying the patch you suggested to CUDA.