build issues with rocm-6.1.0, clang not found, rocm_agend_enumerator not found

[mlohry]

I'm running into a sequence build issues on release v2024.02.1 on ubuntu 22.04, cmake 3.24.2, with rocm-6.1.0 (might be nice to add HIP build instructions to the docs.)

First try,

cmake -DCMAKE_BUILD_TYPE=Release -DENABLE_HIP=ON -DENABLE_OPENMP=OFF -DENABLE_CUDA=OFF -DCMAKE_HIP_ARCHITECTURES=gfx906 ..

CMake Error can't find hip-config.cmake. Why doesn't the build know where to look for the default case of installation in /opt/rocm?

So I add -DCMAKE_PREFIX_PATH=/opt/rocm/lib/cmake, leading to

External rocPRIM not found, ROCPRIM_DIR=

So I add -DROCPRIM_DIR=/opt/rocm and cmake passes, generating the build files.

Run make, and get the error c++: error: unrecognized command-line option ‘--rocm-path=/opt/rocm/lib/cmake’ so I assume this means I must use the hipcc compiler. I add -DCMAKE_CXX_COMPILER=hipcc -DCMAKE_C_COMPILER=hipcc to cmake, and run make which leads to:

sh: 1: /opt/rocm/lib/cmake/llvm/bin/clang: not found
Can't exec "/opt/rocm/lib/cmake/bin/rocm_agent_enumerator": No such file or directory at /usr/bin//hipcc.pl line 488.

And I'm stuck here. /opt/rocm/lib/cmake/llvm/bin/clang doesn't exist, it's in /opt/rocm/lib/llvm/bin/clang. /opt/rocm/lib/cmake/bin/rocm_agent_enumerator also doesn't exist, it's in /opt/rocm/bin/rocm_agent_enumerator. If I change the cmake prefix path to -DCMAKE_PREFIX_PATH=/opt/rocm/ it now looks in a different directory, also incorrect:

sh: 1: /opt/rocm/include/llvm/bin/clang: not found
Can't exec "/opt/rocm/include/bin/rocm_agent_enumerator": No such file or directory at /opt/rocm/bin//hipcc.pl line 488.

and this is where I'm currently stuck.

[Rich Hornung]

@mlohry there are scripts for configuring a build with CMake that we use for daily development in https://github.com/LLNL/RAJA/tree/develop/scripts/lc-builds toss4_amdclang.sh and/or toss4_hipcc.sh may be helpful. Not all of the CMake options related to HIP/ROCM may be required currently. However, they were at some point.

[mlohry]

Thanks @rhornung67 that got me working. One thing that threw me is rocm-6.1.0 doesn't have a /hip subdirectory. Looks like they overhauled the directory structure:

$ ls /opt/rocm-5.7.3/
amdgcn  hip      hipblaslt  hipfft   hiprand    hipsparse  include  libexec  miopen  opencl  rocalution  rocfft    rocprim      rocrand    rocsparse  roctracer
bin     hipblas  hipcub     hipfort  hipsolver  hsa        lib      llvm     oam     rccl    rocblas     rocm_smi  rocprofiler  rocsolver  rocthrust  share

$ ls /opt/rocm-6.1.0/
amdgcn  bin  include  lib  libexec  llvm  share


$ ls /opt/rocm-6.1.0/include/
amd_comgr   amd_smi  CL    hip      hipblaslt  hipfft   hipify   hipsolver  hipsparselt  hsa     miopen  rccl        rocblas  rocm-core  rocprim      rocprofiler-register  rocsolver  roctracer  thrust
amd-dbgapi  ck       half  hipblas  hipcub     hipfort  hiprand  hipsparse  hiptensor    hsakmt  oam     rocalution  rocfft   rocm_smi   rocprofiler  rocrand               rocsparse  rocwmma

[Rich Hornung]

Yes. AMD has been changing their directory structure around for a while. Hopefully, it's settled.