Arch Linux Build

Question

Arch Linux Build

adro79 opened this issue a year ago · comments

Hi, I'm looking to build your renderer in Arch Linux.

Note: I disabled the externals so I can use the system dependencies

Build commands:

cmake .. -Wno-dev -DUSD_ROOT=/usr/share/usd -DMDL_ROOT=/mnt/proj/dep/mdl-sdk -DCMAKE_BUILD_TYPE=Release -DCMAKE_INSTALL_PREFIX=/usr/share/usd -DCMAKE_PREFIX_PATH=/usr
cmake --build . -j2 --target hdGatling gatling --config Release

I have two errors, one building from gcc 13 and the other from clang 15:

GCC

[ 98%] Building CXX object src/hdGatling/CMakeFiles/hdGatling.dir/Tokens.cpp.o
In file included from /usr/include/c++/13.2.1/ext/hash_set:60,
                 from /usr/share/usd/include/pxr/base/tf/hashset.h:39,
                 from /usr/share/usd/include/pxr/base/tf/token.h:37,
                 from /usr/share/usd/include/pxr/base/tf/staticTokens.h:82,
                 from /mnt/proj/dev/gatling/src/hdGatling/Tokens.h:20,
                 from /mnt/proj/dev/gatling/src/hdGatling/Tokens.cpp:18:
/usr/include/c++/13.2.1/backward/backward_warning.h:32:2: warning: #warning This file includes at least one deprecated or antiquated header which may be removed without further notice at a future date. Please use a non-deprecated interface with equivalent functionality instead. For a listing of replacement headers and interfaces, consult the file backward_warning.h. To disable this warning use -Wno-deprecated. [-Wcpp]
   32 | #warning \
      |  ^~~~~~~
In file included from /usr/include/c++/13.2.1/ext/hash_set:64:
/usr/include/c++/13.2.1/backward/hashtable.h: In instantiation of ‘__gnu_cxx::hashtable<_Val, _Key, _HashFcn, _ExtractKey, _EqualKey, _Alloc>::size_type __gnu_cxx::hashtable<_Val, _Key, _HashFcn, _ExtractKey, _EqualKey, _Alloc>::_M_bkt_num_key(const key_type&, std::size_t) const [with _Val = std::pair<const std::__cxx11::basic_string<char>, unsigned int>; _Key = std::__cxx11::basic_string<char>; _HashFcn = __gnu_cxx::hash<std::__cxx11::basic_string<char> >; _ExtractKey = std::_Select1st<std::pair<const std::__cxx11::basic_string<char>, unsigned int> >; _EqualKey = std::equal_to<std::__cxx11::basic_string<char> >; _Alloc = std::allocator<unsigned int>; size_type = long unsigned int; key_type = std::__cxx11::basic_string<char>; std::size_t = long unsigned int]’:
/usr/include/c++/13.2.1/backward/hashtable.h:596:30:   required from ‘__gnu_cxx::hashtable<_Val, _Key, _HashFcn, _ExtractKey, _EqualKey, _Alloc>::size_type __gnu_cxx::hashtable<_Val, _Key, _HashFcn, _ExtractKey, _EqualKey, _Alloc>::_M_bkt_num_key(const key_type&) const [with _Val = std::pair<const std::__cxx11::basic_string<char>, unsigned int>; _Key = std::__cxx11::basic_string<char>; _HashFcn = __gnu_cxx::hash<std::__cxx11::basic_string<char> >; _ExtractKey = std::_Select1st<std::pair<const std::__cxx11::basic_string<char>, unsigned int> >; _EqualKey = std::equal_to<std::__cxx11::basic_string<char> >; _Alloc = std::allocator<unsigned int>; size_type = long unsigned int; key_type = std::__cxx11::basic_string<char>]’
/usr/include/c++/13.2.1/backward/hashtable.h:519:32:   required from ‘__gnu_cxx::hashtable<_Val, _Key, _HashFcn, _ExtractKey, _EqualKey, _Alloc>::iterator __gnu_cxx::hashtable<_Val, _Key, _HashFcn, _ExtractKey, _EqualKey, _Alloc>::find(const key_type&) [with _Val = std::pair<const std::__cxx11::basic_string<char>, unsigned int>; _Key = std::__cxx11::basic_string<char>; _HashFcn = __gnu_cxx::hash<std::__cxx11::basic_string<char> >; _ExtractKey = std::_Select1st<std::pair<const std::__cxx11::basic_string<char>, unsigned int> >; _EqualKey = std::equal_to<std::__cxx11::basic_string<char> >; _Alloc = std::allocator<unsigned int>; iterator = __gnu_cxx::hashtable<std::pair<const std::__cxx11::basic_string<char>, unsigned int>, std::__cxx11::basic_string<char>, __gnu_cxx::hash<std::__cxx11::basic_string<char> >, std::_Select1st<std::pair<const std::__cxx11::basic_string<char>, unsigned int> >, std::equal_to<std::__cxx11::basic_string<char> >, std::allocator<unsigned int> >::iterator; key_type = std::__cxx11::basic_string<char>]’
/usr/include/c++/13.2.1/ext/hash_map:213:26:   required from ‘__gnu_cxx::hash_map<_Key, _Tp, _HashFn, _EqualKey, _Alloc>::iterator __gnu_cxx::hash_map<_Key, _Tp, _HashFn, _EqualKey, _Alloc>::find(const key_type&) [with _Key = std::__cxx11::basic_string<char>; _Tp = unsigned int; _HashFn = __gnu_cxx::hash<std::__cxx11::basic_string<char> >; _EqualKey = std::equal_to<std::__cxx11::basic_string<char> >; _Alloc = std::allocator<unsigned int>; iterator = __gnu_cxx::hashtable<std::pair<const std::__cxx11::basic_string<char>, unsigned int>, std::__cxx11::basic_string<char>, __gnu_cxx::hash<std::__cxx11::basic_string<char> >, std::_Select1st<std::pair<const std::__cxx11::basic_string<char>, unsigned int> >, std::equal_to<std::__cxx11::basic_string<char> >, std::allocator<unsigned int> >::iterator; key_type = std::__cxx11::basic_string<char>]’
/usr/share/usd/include/pxr/base/tf/hashmap.h:118:60:   required from ‘pxrInternal_v0_23__pxrReserved__::TfHashMap<Key, Mapped, HashFn, EqualKey, Alloc>::iterator pxrInternal_v0_23__pxrReserved__::TfHashMap<Key, Mapped, HashFn, EqualKey, Alloc>::find(const key_type&) [with Key = std::__cxx11::basic_string<char>; Mapped = unsigned int; HashFn = __gnu_cxx::hash<std::__cxx11::basic_string<char> >; EqualKey = std::equal_to<std::__cxx11::basic_string<char> >; Alloc = std::allocator<unsigned int>; iterator = __gnu_cxx::hashtable<std::pair<const std::__cxx11::basic_string<char>, unsigned int>, std::__cxx11::basic_string<char>, __gnu_cxx::hash<std::__cxx11::basic_string<char> >, std::_Select1st<std::pair<const std::__cxx11::basic_string<char>, unsigned int> >, std::equal_to<std::__cxx11::basic_string<char> >, std::allocator<unsigned int> >::iterator; key_type = std::__cxx11::basic_string<char>]’
/mnt/proj/dev/gatling/src/hdGatling/RenderPass.cpp:468:50:   required from here
/usr/include/c++/13.2.1/backward/hashtable.h:604:23: error: no match for call to ‘(const __gnu_cxx::hashtable<std::pair<const std::__cxx11::basic_string<char>, unsigned int>, std::__cxx11::basic_string<char>, __gnu_cxx::hash<std::__cxx11::basic_string<char> >, std::_Select1st<std::pair<const std::__cxx11::basic_string<char>, unsigned int> >, std::equal_to<std::__cxx11::basic_string<char> >, std::allocator<unsigned int> >::hasher {aka const __gnu_cxx::hash<std::__cxx11::basic_string<char> >}) (const __gnu_cxx::hashtable<std::pair<const std::__cxx11::basic_string<char>, unsigned int>, std::__cxx11::basic_string<char>, __gnu_cxx::hash<std::__cxx11::basic_string<char> >, std::_Select1st<std::pair<const std::__cxx11::basic_string<char>, unsigned int> >, std::equal_to<std::__cxx11::basic_string<char> >, std::allocator<unsigned int> >::key_type&)’
  604 |       { return _M_hash(__key) % __n; }
      |                ~~~~~~~^~~~~~~
make[3]: *** [src/hdGatling/CMakeFiles/hdGatling.dir/build.make:258: src/hdGatling/CMakeFiles/hdGatling.dir/RenderPass.cpp.o] Error 1
make[2]: *** [CMakeFiles/Makefile2:359: src/hdGatling/CMakeFiles/hdGatling.dir/all] Error 2
make[1]: *** [CMakeFiles/Makefile2:366: src/hdGatling/CMakeFiles/hdGatling.dir/rule] Error 2
make: *** [Makefile:234: hdGatling] Error 2

Clang

[100%] Linking CXX shared library ../../bin/hdGatling.so
/usr/bin/ld: ../../bin/libgi.a: error adding symbols: file format not recognized
clang-15: error: linker command failed with exit code 1 (use -v to see invocation)
make[3]: *** [src/hdGatling/CMakeFiles/hdGatling.dir/build.make:378: bin/hdGatling.so] Error 1
make[2]: *** [CMakeFiles/Makefile2:359: src/hdGatling/CMakeFiles/hdGatling.dir/all] Error 2
make[1]: *** [CMakeFiles/Makefile2:366: src/hdGatling/CMakeFiles/hdGatling.dir/rule] Error 2
make: *** [Makefile:234: hdGatling] Error 2

Pablo Delgado · Answer 1 · Sat Aug 12 2023 21:18:57 GMT+0800 (China Standard Time)

Hello! Sorry for these issues; I've been meaning to set up a CI but it's not trivial (yet) how to automatically get MDL SDK binaries.

I'll have a look at the GCC issue when I have time. As for Clang - it seems to be fine with AppleClang. Can you verify that the error message has nothing to do with the previous GCC attempt (e.g. clear the build directory)?

adro79 · Answer 2 · Mon Aug 14 2023 20:35:31 GMT+0800 (China Standard Time)

Hi, I can confirm that I deleted the build folder on each try.

Pablo Delgado · Answer 3 · Fri Aug 18 2023 22:32:38 GMT+0800 (China Standard Time)

Hi, I've fixed the GCC issues on the dev branch (tested with GCC 12.3.0 on Ubuntu 23.04).

adro79 · Answer 4 · Sun Aug 20 2023 00:05:43 GMT+0800 (China Standard Time)

That fixed the GCC issue, thanks!

Now I'm facing this issue when linking hdGatling.so with submodules. I tried both in Arch and Ubuntu 22.04 LTS

Arch:

/usr/bin/ld: bin/libturbojpeg.a(turbojpeg.c.o): relocation R_X86_64_TPOFF32 against `errStr' can not be used when making a shared object; recompile with -fPIC
/usr/bin/ld: failed to set dynamic section sizes: bad value

Ubuntu:

[ 99%] Linking CXX shared library ../../bin/hdGatling.so
/usr/bin/ld: ../../bin/libturbojpeg.a(turbojpeg.c.o): relocation R_X86_64_TPOFF32 against `errStr' can not be used when making a shared object; recompile with -fPIC
/usr/bin/ld: failed to set dynamic section sizes: bad value
collect2: error: ld returned 1 exit status

However, if I edit the files to use system dependencies I can build Gatling but doesn't start.

gatling /mnt/proj/dev/usd-assets/cornell-box/mtlx-ref/cornell-box.usda test.png

USD scene loaded (0.221s)
rebuilding shader cache
material count: 5
creating shader cache..
error in /mnt/data/Applications/AUR/usd-gatling/src/gatling/src/cgpu/src/cgpu.cpp:982: failed to reflect shader
error in /mnt/data/Applications/AUR/usd-gatling/src/gatling/src/cgpu/src/cgpu.cpp:998: invalid resource handle
Coding Error: in _Execute at line 750 of /mnt/data/Applications/AUR/usd-gatling/src/gatling/src/hdGatling/RenderPass.cpp -- Failed verification: ' m_shaderCache ' -- Unable to create shader cache
rebuilding geom cache
instance count: 16
creating geom cache..
[1]    145434 segmentation fault (core dumped)  gatling /mnt/proj/dev/usd-assets/cornell-box/mtlx-ref/cornell-box.usda test.png

usdview /mnt/proj/dev/usd-assets/cornell-box/mtlx-ref/cornell-box.usda

Generation failed: surface shader not found

adro79 · Answer 5 · Sun Aug 20 2023 00:09:26 GMT+0800 (China Standard Time)

These are the changes I applied to be able to build:
gatling.patch.txt

Pablo Delgado · Answer 6 · Sun Aug 20 2023 02:12:33 GMT+0800 (China Standard Time)

Thanks for the patch. I don't/can't really support all versions of external dependencies, so incompatibilities are expected. I strongly advise to use the submodules instead.

I should have fixed the -fPIC error from above; perhaps run git fetch --all && git reset --hard origin/dev?

error in /mnt/data/Applications/AUR/usd-gatling/src/gatling/src/cgpu/src/cgpu.cpp:982: failed to reflect shader

means that something with SPIRV-Reflect went wrong. Perhaps this patch is missing: KhronosGroup/SPIRV-Reflect#191

Could you delete the extern folder and run git checkout dev extern && git submodule update --init --recursive again?
Alternatively, a fresh clone or commenting out cgpu.cpp:559 could work?

adro79 · Answer 7 · Wed Aug 23 2023 01:15:26 GMT+0800 (China Standard Time)

Hi, thanks for the suggestion but seems that I still have problems building with submodules:

[100%] Linking CXX shared library ../../bin/hdGatling.so
/usr/bin/ld: ../../bin/libOpenEXR-3_1.a(ImfIDManifest.cpp.o): in function `Imf_3_1::(anonymous namespace)::catString(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&) [clone .constprop.0]':
ImfIDManifest.cpp:(.text+0x506b): undefined reference to `std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::_M_assign(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)'
/usr/bin/ld: ../../bin/libOpenEXR-3_1.a(ImfIDManifest.cpp.o): in function `Imf_3_1::IDManifest::ChannelGroupManifest::setComponents(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&)':
ImfIDManifest.cpp:(.text+0x569f): undefined reference to `std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::_M_assign(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)'
/usr/bin/ld: ../../bin/libOpenEXR-3_1.a(ImfIDManifest.cpp.o): in function `Imf_3_1::IDManifest::ChannelGroupManifest::insert(unsigned long, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)':
ImfIDManifest.cpp:(.text+0x71ab): undefined reference to `std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::_M_assign(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)'
/usr/bin/ld: ../../bin/libOpenEXR-3_1.a(ImfIDManifest.cpp.o): in function `Imf_3_1::IDManifest::serialize(std::vector<char, std::allocator<char> >&) const':
ImfIDManifest.cpp:(.text+0x80aa): undefined reference to `std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::_M_assign(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)'
/usr/bin/ld: /opt/materialx/lib/libMaterialXGenMdl.a(MdlShaderGenerator.cpp.o): in function `MaterialX_v1_38_7::MdlShaderGenerator::emitShaderInputs(MaterialX_v1_38_7::VariableBlock const&, MaterialX_v1_38_7::ShaderStage&) const':
(.text+0x2d1d): undefined reference to `std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::_M_assign(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)'
/usr/bin/ld: /opt/materialx/lib/libMaterialXGenMdl.a(MdlShaderGenerator.cpp.o):(.text+0x2d58): more undefined references to `std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::_M_assign(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)' follow
collect2: error: ld returned 1 exit status
make[2]: *** [src/hdGatling/CMakeFiles/hdGatling.dir/build.make:3
![test](https://github.com/pablode/gatling/assets/57686179/c17cf31a-3595-4ef0-a42c-f0fab989862e)
98: bin/hdGatling.so] Error 1
make[1]: *** [CMakeFiles/Makefile2:4914: src/hdGatling/CMakeFiles/hdGatling.dir/all] Error 2
make: *** [Makefile:136: all] Error 2

On the other hand, the version without submodules is able to render using the command line 🎉

However when I raise the samples this happens:

USD scene loaded (0.002s)
rebuilding shader cache
material count: 10
creating shader cache..
creating RT pipeline..
rebuilding geom cache
instance count: 26
creating geom cache..
total geom buffer size: 23.65MiB
> 2.63MiB faces
> 21.02MiB vertices
recreating output buffer with size 4096x4096 (256.00MiB)
error in /mnt/data/Applications/AUR/usd-gatling/src/gatling/src/cgpu/src/cgpu.cpp:3339: failed to wait for fence
Coding Error: in _Execute at line 805 of /mnt/data/Applications/AUR/usd-gatling/src/gatling/src/hdGatling/RenderPass.cpp -- Failed verification: ' result == GI_OK ' -- Unable to render scene.
Rendering finished (7.013s)
Wrote image (0.951s)
error in /mnt/data/Applications/AUR/usd-gatling/src/gatling/src/cgpu/src/cgpu.cpp:3339: failed to wait for fence

usdview still fails to load gatling:
Generation failed: surface shader not found

Pablo Delgado · Answer 8 · Fri Aug 25 2023 15:27:57 GMT+0800 (China Standard Time)

That's great to hear!

The failed to wait for fence error with high sample counts is probably due to a watchdog timeout. Perhaps disable it?

Generation failed: surface shader not found - it sounds to me that the MaterialX libraries were not loaded. Since the standalone also uses Hydra, I suspect that this is a USD problem. Perhaps set PXR_MTLX_STDLIB_SEARCH_PATHS to <USD_INSTALL>/libraries?

Pablo Delgado · Answer 9 · Sun Aug 27 2023 03:51:45 GMT+0800 (China Standard Time)

I have identified and fixed the clang issues; they were caused by LTO (link time optimization).

Pablo Delgado · Answer 10 · Sat Nov 18 2023 18:02:50 GMT+0800 (China Standard Time)

Hi, I've added Linux builds to the CI, so compilation should work in general. Regarding the surface shader not found issue - USD doesn't like being moved around, perhaps that happened here. PXR_MTLX_STDLIB_SEARCH_PATHS from above should help with the discovery of MaterialX's libraries folder here.

Going to close this issue, let me know if there's trouble!