[Question] Getting "failed to open libnvrtc-builtins.so.12.2."

[Hugo Phibbs]

Hi,

For some reason arrayfire isn't working with my nvidia GPU. I'm trying to do this with cmake.

I have the cuda toolkit correctly installed, and followed the linux linux install instructions found here https://arrayfire.org/docs/installing.htm#gsc.tab=0.

The error:

In function compileModule
In file src/backend/cuda/compile_module.cpp:297
NVRTC Error(7): NVRTC_ERROR_BUILTIN_OPERATION_FAILURE
Log: 
nvrtc: error: failed to open libnvrtc-builtins.so.12.2.
  Make sure that libnvrtc-builtins.so.12.2 is installed correctly.

my cmake:

cmake_minimum_required(VERSION 3.27)
project(...)

set(CMAKE_CXX_STANDARD 17)

set(CUDA_TOOLKIT_ROOT_DIR "/usr/local/cuda-11.3")
find_package(CUDAToolkit 11.3 REQUIRED)

include_directories(
        ${PROJECT_SOURCE_DIR}/lib/eigen-3.4.0
        ${CUDA_TOOLKIT_ROOT_DIR}/include
        ${ArrayFire_INCLUDE_DIRS}
        ${gtest_SOURCE_DIR}/include ${gtest_SOURCE_DIR}
)

link_directories(
        ${CUDA_TOOLKIT_ROOT_DIR}/lib64
)

set(ArrayFire_DIR /opt/ArrayFire-3.9.0-Linux/share/ArrayFire/cmake)
find_package(ArrayFire REQUIRED)

if(ArrayFire_CUDA_FOUND)
    add_definitions(-DAF_CUDA)
elseif(ArrayFire_OpenCL_FOUND)
    add_definitions(-DAF_OPENCL)
elseif(ArrayFire_CPU_FOUND)
    add_definitions(-DAF_CPU)
endif()

add_subdirectory(lib/googletest)

add_executable(${PROJECT_NAME}
   ... my files
)

add_executable(run_gs_dbscan_tests
 ... my files
)

set(CUDA_LIBS
        libcudart.so
        libcublas.so
        libcusolver.so
)

target_link_libraries(run_gs_dbscan_tests gtest gtest_main ${CUDA_LIBS} ${ArrayFire_LIBRARIES})
target_link_libraries(${PROJECT_NAME} PRIVATE ${CUDA_LIBS} ${ArrayFire_LIBRARIES})

nvidia-smi:

Mon May 20 12:20:31 2024       
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 550.54.15              Driver Version: 550.54.15      CUDA Version: 12.4     |
|-----------------------------------------+------------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id          Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |
|                                         |                        |               MIG M. |
|=========================================+========================+======================|
|   0  NVIDIA GeForce RTX 3090        Off |   00000000:01:00.0 Off |                  N/A |
|  0%   50C    P8             40W /  390W |      16MiB /  24576MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+
|   1  NVIDIA GeForce RTX 3090        Off |   00000000:4A:00.0 Off |                  N/A |
|  0%   49C    P8             37W /  390W |     628MiB /  24576MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+
                                                                                         
+-----------------------------------------------------------------------------------------+
| Processes:                                                                              |
|  GPU   GI   CI        PID   Type   Process name                              GPU Memory |
|        ID   ID                                                               Usage      |
|=========================================================================================|
|    0   N/A  N/A      1994      G   /usr/lib/xorg/Xorg                              4MiB |
|    0   N/A  N/A      2644      G   /usr/lib/xorg/Xorg                              4MiB |
|    1   N/A  N/A      1994      G   /usr/lib/xorg/Xorg                            150MiB |
|    1   N/A  N/A      2644      G   /usr/lib/xorg/Xorg                            426MiB |
|    1   N/A  N/A      2812      G   /usr/bin/gnome-shell                           24MiB |
+-----------------------------------------------------------------------------------------+

nvcc --version:

nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2021 NVIDIA Corporation
Built on Mon_May__3_19:15:13_PDT_2021
Cuda compilation tools, release 11.3, V11.3.109
Build cuda_11.3.r11.3/compiler.29920130_0

lmk if you guys need more info.

thanks v much

[Edwin Lester Solís Fuentes]

From your error log and nvidia-smi output, it seems you compiled/installed ArrayFire for cuda 12.2, but you have installed cuda 12.4. ArrayFire is looking for libnvrtc-builtins.so.12.2, but you likely have libnvrtc-builtins.so.12.4.

If you have installed Cuda in the default path, you can verify this by running ls /usr/local/cuda/lib64 and ls /usr/local/cuda/bin and see the contents of the cuda installation.

To solve your issue, you may recompile ArrayFire from source for Cuda 12.4 by specifying the CMAKE_CUDA_COMPILER location.

PS. It seems you have multiple versions of cuda installed as nvcc --version is outputting Cuda 11.3, make sure you use the same Cuda version as the one ArrayFire is built if you plan to use Cuda interoperability.

Reopen the issue, if you have any other questions or find any other problems