Failed to build unit tests in cachelib
bonii opened this issue · comments
Describe the bug
I failed to build the Cachelib Unit tests as described in the testing guide. I get a linker error with folly, note that the regular build works.
To Reproduce
Steps to reproduce the behavior:
sudo ./contrib/build.sh -j -T
Expected behavior
The build succeeds
** Observed Behavior **
Submodule path 'cachelib/external/fbthrift': checked out 'ee5afbb82d2b0fbd5082dd3bdbee7315456db49f'
Submodule path 'cachelib/external/fizz': checked out 'd7bcaa8d7272ba152d80e2ce5e0dd1f3a030995c'
Submodule path 'cachelib/external/folly': checked out 'e3b9b1e936e1d27908a18ee27c62274db87ca5ab'
Submodule path 'cachelib/external/wangle': checked out '379c9092f06420d010509b8f33bcdcdf75e00d0c'
Entering 'cachelib/external/fbthrift'
Entering 'cachelib/external/fizz'
Entering 'cachelib/external/folly'
Entering 'cachelib/external/wangle'
HEAD is now at 379c9092 Updating submodules
HEAD is now at d7bcaa8d Updating submodules
-- other platform:
-- Found gflags from package config /home/bonii/projects/samsung/Cachelib/opt/cachelib/lib/cmake/gflags/gflags-config.cmake
-- Found gflags as a dependency of glog::glog, include=/home/bonii/projects/samsung/Cachelib/opt/cachelib/include, libs=gflags_shared
-- Found libevent: /usr/lib/x86_64-linux-gnu/libevent.so
-- Found LZ4: /usr/lib/x86_64-linux-gnu/liblz4.so
CMake Warning (dev) at /usr/share/cmake-3.25/Modules/FindPackageHandleStandardArgs.cmake:438 (message):
The package name passed to `find_package_handle_standard_args` (ZSTD) does
not match the name of the calling package (Zstd). This can lead to
problems in calling code that expects `find_package` result variables
(e.g., `_FOUND`) to follow a certain pattern.
Call Stack (most recent call first):
CMake/FindZstd.cmake:32 (FIND_PACKAGE_HANDLE_STANDARD_ARGS)
CMake/folly-deps.cmake:114 (find_package)
CMakeLists.txt:141 (include)
This warning is for project developers. Use -Wno-dev to suppress it.
-- Found Zstd: /home/bonii/projects/samsung/Cachelib/opt/cachelib/lib/libzstd.so
CMake Warning (dev) at /usr/share/cmake-3.25/Modules/FindPackageHandleStandardArgs.cmake:438 (message):
The package name passed to `find_package_handle_standard_args` (SNAPPY)
does not match the name of the calling package (Snappy). This can lead to
problems in calling code that expects `find_package` result variables
(e.g., `_FOUND`) to follow a certain pattern.
Call Stack (most recent call first):
CMake/FindSnappy.cmake:31 (FIND_PACKAGE_HANDLE_STANDARD_ARGS)
CMake/folly-deps.cmake:121 (find_package)
CMakeLists.txt:141 (include)
This warning is for project developers. Use -Wno-dev to suppress it.
CMake Warning (dev) at /usr/share/cmake-3.25/Modules/FindPackageHandleStandardArgs.cmake:438 (message):
The package name passed to `find_package_handle_standard_args` (LIBDWARF)
does not match the name of the calling package (LibDwarf). This can lead
to problems in calling code that expects `find_package` result variables
(e.g., `_FOUND`) to follow a certain pattern.
Call Stack (most recent call first):
CMake/FindLibDwarf.cmake:25 (FIND_PACKAGE_HANDLE_STANDARD_ARGS)
CMake/folly-deps.cmake:128 (find_package)
CMakeLists.txt:141 (include)
This warning is for project developers. Use -Wno-dev to suppress it.
CMake Warning (dev) at /usr/share/cmake-3.25/Modules/FindPackageHandleStandardArgs.cmake:438 (message):
The package name passed to `find_package_handle_standard_args` (LIBIBERTY)
does not match the name of the calling package (Libiberty). This can lead
to problems in calling code that expects `find_package` result variables
(e.g., `_FOUND`) to follow a certain pattern.
Call Stack (most recent call first):
CMake/FindLibiberty.cmake:22 (FIND_PACKAGE_HANDLE_STANDARD_ARGS)
CMake/folly-deps.cmake:132 (find_package)
CMakeLists.txt:141 (include)
This warning is for project developers. Use -Wno-dev to suppress it.
CMake Warning (dev) at /usr/share/cmake-3.25/Modules/FindPackageHandleStandardArgs.cmake:438 (message):
The package name passed to `find_package_handle_standard_args` (LIBAIO)
does not match the name of the calling package (LibAIO). This can lead to
problems in calling code that expects `find_package` result variables
(e.g., `_FOUND`) to follow a certain pattern.
Call Stack (most recent call first):
CMake/FindLibAIO.cmake:22 (FIND_PACKAGE_HANDLE_STANDARD_ARGS)
CMake/folly-deps.cmake:136 (find_package)
CMakeLists.txt:141 (include)
This warning is for project developers. Use -Wno-dev to suppress it.
-- Could NOT find LIBAIO (missing: LIBAIO_LIBRARY LIBAIO_INCLUDE_DIR)
CMake Warning (dev) at /usr/share/cmake-3.25/Modules/FindPackageHandleStandardArgs.cmake:438 (message):
The package name passed to `find_package_handle_standard_args` (LIBURING)
does not match the name of the calling package (LibUring). This can lead
to problems in calling code that expects `find_package` result variables
(e.g., `_FOUND`) to follow a certain pattern.
Call Stack (most recent call first):
CMake/FindLibUring.cmake:22 (FIND_PACKAGE_HANDLE_STANDARD_ARGS)
CMake/folly-deps.cmake:140 (find_package)
CMakeLists.txt:141 (include)
This warning is for project developers. Use -Wno-dev to suppress it.
CMake Warning (dev) at /usr/share/cmake-3.25/Modules/FindPackageHandleStandardArgs.cmake:438 (message):
The package name passed to `find_package_handle_standard_args` (LIBSODIUM)
does not match the name of the calling package (Libsodium). This can lead
to problems in calling code that expects `find_package` result variables
(e.g., `_FOUND`) to follow a certain pattern.
Call Stack (most recent call first):
CMake/FindLibsodium.cmake:22 (FIND_PACKAGE_HANDLE_STANDARD_ARGS)
CMake/folly-deps.cmake:144 (find_package)
CMakeLists.txt:141 (include)
This warning is for project developers. Use -Wno-dev to suppress it.
-- Found Libsodium: /usr/lib/x86_64-linux-gnu/libsodium.so
CMake Warning (dev) at /usr/share/cmake-3.25/Modules/FindPackageHandleStandardArgs.cmake:438 (message):
The package name passed to `find_package_handle_standard_args` (LIBUNWIND)
does not match the name of the calling package (LibUnwind). This can lead
to problems in calling code that expects `find_package` result variables
(e.g., `_FOUND`) to follow a certain pattern.
Call Stack (most recent call first):
build/fbcode_builder/CMake/FindLibUnwind.cmake:22 (FIND_PACKAGE_HANDLE_STANDARD_ARGS)
CMake/folly-deps.cmake:156 (find_package)
CMakeLists.txt:141 (include)
This warning is for project developers. Use -Wno-dev to suppress it.
-- Setting FOLLY_USE_SYMBOLIZER: ON
-- Setting FOLLY_HAVE_ELF: 1
-- Setting FOLLY_HAVE_DWARF: TRUE
-- GCC has support for C++ coroutines, setting flag for Folly build.
-- arch x86_64-linux-gnu matches x86_64, building SSE4.2 version of base64
-- arch x86_64-linux-gnu matches x86_64, setting SSE2/AVX2 compile flags for LtHash SIMD code
-- compiler has flag pclmul, setting compile flag for /home/bonii/projects/samsung/Cachelib/cachelib/external/folly/folly/hash/detail/ChecksumDetail.cpp;/home/bonii/projects/samsung/Cachelib/cachelib/external/folly/folly/hash/detail/Crc32CombineDetail.cpp;/home/bonii/projects/samsung/Cachelib/cachelib/external/folly/folly/hash/detail/Crc32cDetail.cpp
-- Configuring done
-- Generating done
-- Build files have been written to: /home/bonii/projects/samsung/Cachelib/build-folly
[ 94%] Built target folly_base
[ 94%] Built target folly
[ 95%] Built target logging_example_lib
[ 96%] Built target folly_test_util
[ 97%] Built target folly_exception_tracer_base
[ 97%] Built target follybenchmark
[ 97%] Linking CXX executable logging_example
[ 98%] Built target folly_exception_tracer
[ 99%] Built target folly_exception_counter
/usr/bin/ld: ../../../libfolly.so.0.58.0-dev: undefined reference to `google::InitVLOG3__(int**, int*, char const*, int)'
/usr/bin/ld: ../../../libfolly.so.0.58.0-dev: undefined reference to `google::LogMessage::LogMessage(char const*, int, int, unsigned long, void (google::LogMessage::*)())'
/usr/bin/ld: ../../../libfolly.so.0.58.0-dev: undefined reference to `google::ErrnoLogMessage::ErrnoLogMessage(char const*, int, int, unsigned long, void (google::LogMessage::*)())'
collect2: error: ld returned 1 exit status
make[2]: *** [folly/logging/example/CMakeFiles/logging_example.dir/build.make:124: folly/logging/example/logging_example] Error 1
make[1]: *** [CMakeFiles/Makefile2:331: folly/logging/example/CMakeFiles/logging_example.dir/all] Error 2
make: *** [Makefile:136: all] Error 2
build-package.sh: error: make failed
build.sh: error: failed to build dependency 'folly'
Desktop (please complete the following information):
bonii@razorblade:~/projects/samsung/Cachelib$ cat /etc/os-release
PRETTY_NAME="Debian GNU/Linux 11 (bullseye)"
NAME="Debian GNU/Linux"
VERSION_ID="11"
VERSION="11 (bullseye)"
VERSION_CODENAME=bullseye
ID=debian
HOME_URL="https://www.debian.org/"
SUPPORT_URL="https://www.debian.org/support"
BUG_REPORT_URL="https://bugs.debian.org/"
bonii@razorblade:~/projects/samsung/Cachelib$ uname -a
Linux razorblade 6.1.0-5-amd64 #1 SMP PREEMPT_DYNAMIC Debian 6.1.12-1 (2023-02-15) x86_64 GNU/Linux
Additional context
Add any other context about the problem here.
The command below to build and install cachelib succeeds.
sudo ./contrib/build.sh -d -j -v
The alternate command below to build cachelib unit tests succeeds.
sudo ./contrib/build-package.sh -j -t cachelib
However, there are no executables under /opt/cachelib/tests
bonii@razorblade:~/projects/samsung/Cachelib$ ls ./opt/cachelib/tests
ls: cannot access './opt/cachelib/tests': No such file or directory
The command sudo ./contrib/build-package.sh -j -t cachelib builds tests under ./build-cachelib/.. but they are not executable
bonii@razorblade:~/projects/samsung/Cachelib$ find . -name navy-test-LruPolicyTest
./build-cachelib/navy/navy-test-LruPolicyTest
bonii@razorblade:~/projects/samsung/Cachelib$ ./build-cachelib/navy/navy-test-LruPolicyTest
./build-cachelib/navy/navy-test-LruPolicyTest: error while loading shared libraries: libfolly_exception_tracer.so.0.58.0-dev: cannot open shared object file: No such file or directory
Hi @bonii
As you mentioned, it seems that there is some regression in building the tests for folly. So, please ignore the build errors from folly or any other external modules.
For building cachelib tests, you need to pass the install option (-i); i.e., ./contrib/build-package.sh -jitv cachelib
Hi @jaesoo-fb
Thanks a lot for checking the issue and the tip with the install option.
However, I still get the same error while running the test that I did when I built without the install option (as in the original report).
See the output of the following command that tries to run a test installed under /opt/cachelib/tests/
bonii@razorblade:~/projects/samsung/Cachelib$ ./opt/cachelib/tests/cachebench-test-NandWritesTest
./opt/cachelib/tests/cachebench-test-NandWritesTest: error while loading shared libraries: libfolly_exception_tracer_base.so.0.58.0-dev: cannot open shared object file: No such file or directory
Do I need to configure any environment variables prior to running the tests or am I missing some step here?
Hi @bonii,
Oh, well, it looks like the same problem; you need to pass -i option for building other packages too.
Hi @jaesoo-fb ,
I am not sure I quite follow. Could you give me a set of commands to try to ensure I am following what you are suggesting and not doing something wrong.
I am currently doing the following:
./contrib/build.sh -jdv
./contrib/build-package.sh -jdiv folly
./contrib/build-package.sh -jditv cachelib
I get an error while trying to run the cachebench and navy tests
bonii@razorblade:~/projects/samsung/Cachelib$ opt/cachelib/tests/cachebench-test-NandWritesTest
opt/cachelib/tests/cachebench-test-NandWritesTest: error while loading shared libraries: libfolly_exception_tracer_base.so.0.58.0-dev: cannot open shared object file: No such file or directory
bonii@razorblade:~/projects/samsung/Cachelib$ opt/cachelib/tests/navy-test-AllocatorTest
opt/cachelib/tests/navy-test-AllocatorTest: error while loading shared libraries: libfolly_exception_tracer_base.so.0.58.0-dev: cannot open shared object file: No such file or directory
I can run other tests though e.g.
bonii@razorblade:~/projects/samsung/Cachelib$ opt/cachelib/tests/allocator-test-AllocationClassTest
Running main() from /home/bonii/projects/samsung/Cachelib/cachelib/external/googletest/googletest/src/gtest_main.cc
[==========] Running 20 tests from 1 test suite.
[----------] Global test environment set-up.
[----------] 20 tests from AllocationClassTest
[ RUN ] AllocationClassTest.Basic
[ OK ] AllocationClassTest.Basic (3 ms)
[ RUN ] AllocationClassTest.AllocFree
[ OK ] AllocationClassTest.AllocFree (52 ms)
[ RUN ] AllocationClassTest.BadFree
[ OK ] AllocationClassTest.BadFree (53 ms)
[ RUN ] AllocationClassTest.AddReleaseSlab
I can find the missing shared library and explicitly add it to LD_LIBRARY_PATH and then some executables work e.g.,
bonii@razorblade:~/projects/samsung/Cachelib$ find . -name libfolly_exception_tracer_base.so.0.58.0-dev
./opt/cachelib/lib/libfolly_exception_tracer_base.so.0.58.0-dev
./build-folly/folly/experimental/exception_tracer/libfolly_exception_tracer_base.so.0.58.0-dev
bonii@razorblade:~/projects/samsung/Cachelib$ export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:./opt/cachelib/lib/
bonii@razorblade:~/projects/samsung/Cachelib$ ./opt/cachelib/tests/cachebench-test-WorkloadGeneratorTest
Running main() from /home/bonii/projects/samsung/Cachelib/cachelib/external/googletest/googletest/src/gtest_main.cc
[==========] Running 3 tests from 1 test suite.
[----------] Global test environment set-up.
[----------] 3 tests from WorkloadGeneratorTest
[ RUN ] WorkloadGeneratorTest.SimplePiecewiseValueSizes
Created 1,000 keys in 0.00 mins
Generating 0.01M sampled accesses
Generated access patterns in 0.00 mins
[ OK ] WorkloadGeneratorTest.SimplePiecewiseValueSizes (5 ms)
[ RUN ] WorkloadGeneratorTest.SimpleDiscreteValueSizes
Created 1,000 keys in 0.00 mins
Generating 0.01M sampled accesses
Generated access patterns in 0.00 mins
[ OK ] WorkloadGeneratorTest.SimpleDiscreteValueSizes (4 ms)
[ RUN ] WorkloadGeneratorTest.InvalidValueSizes
[ OK ] WorkloadGeneratorTest.InvalidValueSizes (0 ms)
[----------] 3 tests from WorkloadGeneratorTest (10 ms total)
[----------] Global test environment tear-down
[==========] 3 tests from 1 test suite ran. (10 ms total)
[ PASSED ] 3 tests.
However some others fail which I am interested in.
bonii@razorblade:~/projects/samsung/Cachelib$ ./opt/cachelib/tests/cachebench-test-NandWritesTest
ERROR: something wrong with flag 'flagfile' in file '/home/bonii/projects/samsung/Cachelib/cachelib/external/gflags/src/gflags.cc'. One possibility: file '/home/bonii/projects/samsung/Cachelib/cachelib/external/gflags/src/gflags.cc' is being linked both statically and dynamically into this executable.
Am I doing something wrong? I would really like to be able to run all the tests. Can you please give me a sequence of commands that I should execute to make sure the tests work? I wanted to make sure I am not misunderstanding your instructions.
@bonii Re "a sequence of commands that I should execute to make sure the tests work": You could try following the commands (on a clean machine/VM/container, ideally) in the GitHub action that corresponds to your OS: https://github.com/facebook/CacheLib/tree/main/.github/workflows -- personally, I found that helpful.
@bonii I cannot find any issues with the steps and cachebench-test-NandWritesTest just works fine for me with debian-10. Could you try after removing all the artifacts generated?
$ rm -rf build-* opt/
$ ./contrib/build.sh -jv
$ ./contrib/build-package.sh -jitv cachelib
Thanks @jaesoo-fb @wonglkd for the pointers.
I will try out with a clean machine environment. I have been trying on a machine with Debian 11 and a custom kernel. I got the same error on the machine with a Debian 10 docker container. Maybe my environment is tainted. I will try with a clean environment and report back.