libssh2-1.11.0 dumps core during testsuite on FreeBSD 14.0 AMD64
blastwave opened this issue · comments
Description :
libssh2-1.11.0 dumps core during testsuite on FreeBSD 14.0 AMD64
This is a bit of a surprise and I still am not sure what happened. This
is a FreeBSD 14.0 server that runs great. Since the FreeBSD project is
pondering a release of 14.0 sometime this year it makes sense to kick
around code from everywhere. Certainly libssh2 is essential stuff.
hydra$ uname -apKU
FreeBSD hydra 14.0-CURRENT FreeBSD 14.0-CURRENT #2 main-n264010-8df38859d0f9: Fri Jul 7 21:38:34 UTC 2023 [root@hydra:/usr/obj/usr/src/amd64.amd64/sys/GENERIC](mailto:root@hydra:/usr/obj/usr/src/amd64.amd64/sys/GENERIC) amd64 amd64 1400093 1400093
hydra$
The configure seems perfectly fine :
hydra$ ./configure --prefix=/opt/bw \
> --disable-silent-rules --enable-dependency-tracking \
> --enable-shared --enable-static --enable-debug \
> --disable-hidden-symbols --disable-docker-tests \
> --enable-examples-build --without-gnu-ld \
> --with-crypto=openssl --with-libssl-prefix=/opt/bw \
> --with-libgcrypt-prefix=/opt/bw --with-libz-prefix=/opt/bw
checking whether to enable maintainer-specific portions of Makefiles... no
checking whether /usr/local/bin/gmake supports nested variables... yes
checking for sed... /usr/bin/sed
checking for a BSD-compatible install... /usr/bin/install -c
checking whether build environment is sane... yes
checking for a race-free mkdir -p... ./install-sh -c -d
checking for gawk... no
checking for mawk... no
checking for nawk... nawk
checking whether /usr/local/bin/gmake sets $(MAKE)... yes
checking libssh2 version... 1.11.0
checking build system type... x86_64-unknown-freebsd14.0
checking host system type... x86_64-unknown-freebsd14.0
configure: autobuild project... libssh2
configure: autobuild revision... 1.11.0
configure: autobuild hostname... hydra
configure: autobuild timestamp... 20230712-073506
checking whether /usr/local/bin/gmake supports the include directive... yes (GNU style)
checking for gcc... /usr/bin/cc
checking whether the C compiler works... yes
checking for C compiler default output file name... a.out
checking for suffix of executables...
checking whether we are cross compiling... no
checking for suffix of object files... o
checking whether the compiler supports GNU C... yes
checking whether /usr/bin/cc accepts -g... yes
checking for /usr/bin/cc option to enable C11 features... none needed
checking whether /usr/bin/cc understands -c and -o together... yes
checking dependency style of /usr/bin/cc... gcc3
checking if _REENTRANT is already defined... no
checking if _REENTRANT is actually needed... no
checking if _REENTRANT is onwards defined... no
checking for library containing socket... none required
checking for library containing inet_addr... none required
checking for gcc... (cached) /usr/bin/cc
checking whether the compiler supports GNU C... (cached) yes
checking whether /usr/bin/cc accepts -g... (cached) yes
checking for /usr/bin/cc option to enable C11 features... (cached) none needed
checking whether /usr/bin/cc understands -c and -o together... (cached) yes
checking dependency style of /usr/bin/cc... (cached) gcc3
checking for g++... no
checking for c++... c++
checking whether the compiler supports GNU C++... yes
checking whether c++ accepts -g... yes
checking for c++ option to enable C++11 features... none needed
checking dependency style of c++... gcc3
checking whether ln -s works... yes
checking whether /usr/local/bin/gmake sets $(MAKE)... (cached) yes
checking for sshd... /usr/sbin/sshd
checking how to print strings... printf
checking for a sed that does not truncate output... (cached) /usr/bin/sed
checking for grep that handles long lines and -e... /usr/bin/grep
checking for egrep... /usr/bin/grep -E
checking for fgrep... /usr/bin/grep -F
checking for ld used by /usr/bin/cc... /usr/local/bin/x86_64-unknown-freebsd14.0-ld
checking if the linker (/usr/local/bin/x86_64-unknown-freebsd14.0-ld) is GNU ld... yes
checking for BSD- or MS-compatible name lister (nm)... /usr/local/bin/nm -B
checking the name lister (/usr/local/bin/nm -B) interface... BSD nm
checking the maximum length of command line arguments... 393216
checking how to convert x86_64-unknown-freebsd14.0 file names to x86_64-unknown-freebsd14.0 format... func_convert_file_noop
checking how to convert x86_64-unknown-freebsd14.0 file names to toolchain format... func_convert_file_noop
checking for /usr/local/bin/x86_64-unknown-freebsd14.0-ld option to reload object files... -r
checking for file... file
checking for objdump... objdump
checking how to recognize dependent libraries... pass_all
checking for dlltool... no
checking how to associate runtime and link libraries... printf %s\n
checking for ar... ar
checking for archiver @FILE support... no
checking for strip... strip
checking for ranlib... ranlib
checking command to parse /usr/local/bin/nm -B output from /usr/bin/cc object... ok
checking for sysroot... no
checking for a working dd... /bin/dd
checking how to truncate binary pipes... /bin/dd bs=4096 count=1
checking for mt... mt
checking if mt is a manifest tool... no
checking for stdio.h... yes
checking for stdlib.h... yes
checking for string.h... yes
checking for inttypes.h... yes
checking for stdint.h... yes
checking for strings.h... yes
checking for sys/stat.h... yes
checking for sys/types.h... yes
checking for unistd.h... yes
checking for dlfcn.h... yes
checking for objdir... .libs
checking if /usr/bin/cc supports -fno-rtti -fno-exceptions... yes
checking for /usr/bin/cc option to produce PIC... -fPIC -DPIC
checking if /usr/bin/cc PIC flag -fPIC -DPIC works... yes
checking if /usr/bin/cc static flag -static works... yes
checking if /usr/bin/cc supports -c -o file.o... yes
checking if /usr/bin/cc supports -c -o file.o... (cached) yes
checking whether the /usr/bin/cc linker (/usr/local/bin/x86_64-unknown-freebsd14.0-ld) supports shared libraries... yes
checking whether -lc should be explicitly linked in... no
checking dynamic linker characteristics... freebsd14.0 ld.so
checking how to hardcode library paths into programs... immediate
checking whether stripping libraries is possible... yes
checking if libtool supports shared libraries... yes
checking whether to build shared libraries... yes
checking whether to build static libraries... yes
checking how to run the C++ preprocessor... c++ -E
checking for ld used by c++... /usr/local/bin/x86_64-unknown-freebsd14.0-ld
checking if the linker (/usr/local/bin/x86_64-unknown-freebsd14.0-ld) is GNU ld... yes
checking whether the c++ linker (/usr/local/bin/x86_64-unknown-freebsd14.0-ld) supports shared libraries... yes
checking for c++ option to produce PIC... -fPIC -DPIC
checking if c++ PIC flag -fPIC -DPIC works... yes
checking if c++ static flag -static works... yes
checking if c++ supports -c -o file.o... yes
checking if c++ supports -c -o file.o... (cached) yes
checking whether the c++ linker (/usr/local/bin/x86_64-unknown-freebsd14.0-ld) supports shared libraries... yes
checking dynamic linker characteristics... freebsd14.0 ld.so
checking how to hardcode library paths into programs... immediate
checking whether byte ordering is bigendian... no
checking for windres... no
checking for special C compiler options needed for large files... no
checking for _FILE_OFFSET_BITS value needed for large files... no
checking for ld used by GCC... /usr/local/bin/x86_64-unknown-freebsd14.0-ld
checking if the linker (/usr/local/bin/x86_64-unknown-freebsd14.0-ld) is GNU ld... yes
checking for shared library run path origin... done
checking for libssl... yes
checking how to link with libssl... /opt/bw/lib/libssl.so /opt/bw/lib/libcrypto.so -Wl,-rpath -Wl,/opt/bw/lib
checking for libz... yes
checking how to link with libz... /opt/bw/lib/libz.so -Wl,-rpath -Wl,/opt/bw/lib
checking whether to enable pedantic and debug compiler options... checking how to run the C preprocessor... /usr/bin/cc -E
checking if cpp -P is needed... no
yes
checking if compiler is clang... yes
checking if compiler is xlclang... no
checking for icc in use... no
checking clang version... 1600
configure: Added this set of compiler options: -pedantic -Wall -Wextra -Wpointer-arith -Wwrite-strings -Wshadow -Winline -Wnested-externs -Wmissing-declarations -Wmissing-prototypes -Wno-long-long -Wfloat-equal -Wno-multichar -Wsign-compare -Wundef -Wno-format-nonliteral -Wendif-labels -Wstrict-prototypes -Wdeclaration-after-statement -Wcast-align -Wno-system-headers -Wshorten-64-to-32 -Wunused -Wvla -Wshift-sign-overflow -Wbad-function-cast -Wconversion -Wempty-body -Wignored-qualifiers -Wtype-limits -Wno-sign-conversion -Wenum-conversion -Wmissing-variable-declarations -Wunused-const-variable -Wdouble-promotion -Wcomma -Wassign-enum -Wextra-semi-stmt
checking whether to enable hidden symbols in the library... no
checking whether to build example applications... yes
checking for errno.h... yes
checking for fcntl.h... yes
checking for stdio.h... (cached) yes
checking for unistd.h... (cached) yes
checking for sys/param.h... yes
checking for sys/uio.h... yes
checking for sys/select.h... yes
checking for sys/socket.h... yes
checking for sys/ioctl.h... yes
checking for sys/time.h... yes
checking for arpa/inet.h... yes
checking for netinet/in.h... yes
checking for sys/un.h... yes
checking for windows.h... no
checking for poll... yes
checking for gettimeofday... yes
checking for select... yes
checking for strtoll... yes
checking for explicit_bzero... yes
checking for explicit_memset... no
checking for memset_s... yes
checking for snprintf... yes
checking for size_t... yes
checking for working alloca.h... no
checking for alloca... yes
checking for an ANSI C-conforming const... yes
checking for inline... inline
checking non-blocking sockets style... O_NONBLOCK
checking whether to enable compiler warnings as errors... no
checking that generated files are newer than configure... done
configure: creating ./config.status
config.status: creating Makefile
config.status: creating src/Makefile
config.status: creating tests/Makefile
config.status: creating tests/ossfuzz/Makefile
config.status: creating example/Makefile
config.status: creating docs/Makefile
config.status: creating libssh2.pc
config.status: creating src/libssh2_config.h
config.status: executing depfiles commands
config.status: executing libtool commands
configure: summary of build options:
version: 1.11.0
Host type: x86_64-unknown-freebsd14.0
Install prefix: /opt/bw
Compiler: /usr/bin/cc
Compiler flags: -std=iso9899:2011 -m64 -g -fno-fast-math -fno-builtin -Wl,-rpath=/opt/bw/lib,--enable-new-dtags -g -pedantic -Wall -Wextra -Wpointer-arith -Wwrite-strings -Wshadow -Winline -Wnested-externs -Wmissing-declarations -Wmissing-prototypes -Wno-long-long -Wfloat-equal -Wno-multichar -Wsign-compare -Wundef -Wno-format-nonliteral -Wendif-labels -Wstrict-prototypes -Wdeclaration-after-statement -Wcast-align -Wno-system-headers -Wshorten-64-to-32 -Wunused -Wvla -Wshift-sign-overflow -Wbad-function-cast -Wconversion -Wempty-body -Wignored-qualifiers -Wtype-limits -Wno-sign-conversion -Wenum-conversion -Wmissing-variable-declarations -Wunused-const-variable -Wdouble-promotion -Wcomma -Wassign-enum -Wextra-semi-stmt
Library types: Shared=yes, Static=yes
Crypto library: OpenSSL
zlib compression: yes
Clear memory: yes
Debug build: yes
Build examples: yes
Run Docker tests: no
Run sshd tests: yes
Path to sshd: /usr/sbin/sshd (only for self-tests)
hydra$
The OpenSSL in this case is ver 3.1.1 and that passes all tests.
Well the compile of libssh2 seems all fine and dandy. The tests are not all fine :
gmake[4]: Entering directory '/opt/bw/build/libssh2-1.11.0_FreeBSD14_amd64.001/tests'
PASS: mansyntax.sh
PASS: test_simple
FAIL: test_sshd.test 1 - sshd-test_ssh2
FAIL: test_sshd.test 2 - sshd-test_auth_pubkey_ok_ed25519
============================================================================
Testsuite summary for libssh2 -
============================================================================
# TOTAL: 4
# PASS: 2
# SKIP: 0
# XFAIL: 0
# FAIL: 2
# XPASS: 0
# ERROR: 0
============================================================================
See tests/test-suite.log
Please report to [libssh2-devel@lists.haxx.se](mailto:libssh2-devel@lists.haxx.se)
============================================================================
gmake[4]: *** [Makefile:1241: test-suite.log] Error 1
gmake[4]: Leaving directory '/opt/bw/build/libssh2-1.11.0_FreeBSD14_amd64.001/tests'
gmake[3]: *** [Makefile:1349: check-TESTS] Error 2
gmake[3]: Leaving directory '/opt/bw/build/libssh2-1.11.0_FreeBSD14_amd64.001/tests'
gmake[2]: *** [Makefile:1608: check-am] Error 2
gmake[2]: Leaving directory '/opt/bw/build/libssh2-1.11.0_FreeBSD14_amd64.001/tests'
gmake[1]: *** [Makefile:1133: check-recursive] Error 1
gmake[1]: Leaving directory '/opt/bw/build/libssh2-1.11.0_FreeBSD14_amd64.001/tests'
gmake: *** [Makefile:530: check-recursive] Error 1
yikes.
There is even a core file :
hydra$
hydra$ find . | grep 'core'
./tests/test_auth_pubkey_ok.core
hydra$
hydra$
hydra$ file ./tests/test_auth_pubkey_ok.core
./tests/test_auth_pubkey_ok.core: ELF 64-bit LSB core file, x86-64, version 1 (FreeBSD), FreeBSD-style, from '/opt/bw/build/libssh2-1.11.0_FreeBSD14_amd64.001/tests/.libs/test_auth_pubkey_ok', pid=62524
hydra$
Searching around and I see there is NO SUCH THING as test_auth_pubkey_ok
binary anywhere. So that is baffling.
Seems to be everything else under the sun :
hydra$
hydra$ ls -1 tests/.libs/test_auth_pubke*
tests/.libs/test_auth_pubkey_fail
tests/.libs/test_auth_pubkey_ok_dsa
tests/.libs/test_auth_pubkey_ok_ecdsa
tests/.libs/test_auth_pubkey_ok_ecdsa_signed
tests/.libs/test_auth_pubkey_ok_ed25519
tests/.libs/test_auth_pubkey_ok_ed25519_encrypted
tests/.libs/test_auth_pubkey_ok_ed25519_mem
tests/.libs/test_auth_pubkey_ok_rsa
tests/.libs/test_auth_pubkey_ok_rsa_encrypted
tests/.libs/test_auth_pubkey_ok_rsa_openssh
tests/.libs/test_auth_pubkey_ok_rsa_signed
hydra$
I do see a log file of some flavour :
hydra$
hydra$ cat ./tests/test_sshd.log
# sshd executable: '/usr/sbin/sshd' (OpenSSH_9.3, OpenSSL 3.0.9 30 May 2023)
# ssh executable: '/usr/bin/ssh' (OpenSSH_9.3p1, OpenSSL 3.0.9 30 May 2023)
# waiting for sshd...
# waiting for sshd...
# waiting for sshd...
# waiting for sshd...
# waiting for sshd...
# waiting for sshd...
# waiting for sshd...
# waiting for sshd...
Fingerprint: 12 FD AD 1E 3B 31 B1 0B AB B0 0F 2A 8D 1B 9A 62 C3 26 BD 2F
Authentication methods: publickey,keyboard-interactive
Authentication by public key failed!
all done
1..2
./test_sshd.test: line 130: 62524 Segmentation fault (core dumped) "${test}"
not ok 1 - sshd-test_ssh2
FAIL: test_sshd.test 1 - sshd-test_ssh2
not ok 2 - sshd-test_auth_pubkey_ok_ed25519
FAIL: test_sshd.test 2 - sshd-test_auth_pubkey_ok_ed25519
hydra$
The test-suite.log is pretty much the same thing :
hydra$
hydra$ cat ./tests/test-suite.log
=====================================
libssh2 -: tests/test-suite.log
=====================================
# TOTAL: 4
# PASS: 2
# SKIP: 0
# XFAIL: 0
# FAIL: 2
# XPASS: 0
# ERROR: 0
.. contents:: :depth: 2
FAIL: test_sshd
===============
# sshd executable: '/usr/sbin/sshd' (OpenSSH_9.3, OpenSSL 3.0.9 30 May 2023)
# ssh executable: '/usr/bin/ssh' (OpenSSH_9.3p1, OpenSSL 3.0.9 30 May 2023)
# waiting for sshd...
# waiting for sshd...
# waiting for sshd...
# waiting for sshd...
# waiting for sshd...
# waiting for sshd...
# waiting for sshd...
# waiting for sshd...
Fingerprint: 12 FD AD 1E 3B 31 B1 0B AB B0 0F 2A 8D 1B 9A 62 C3 26 BD 2F
Authentication methods: publickey,keyboard-interactive
Authentication by public key failed!
all done
1..2
./test_sshd.test: line 130: 62524 Segmentation fault (core dumped) "${test}"
not ok 1 - sshd-test_ssh2
FAIL: test_sshd.test 1 - sshd-test_ssh2
not ok 2 - sshd-test_auth_pubkey_ok_ed25519
FAIL: test_sshd.test 2 - sshd-test_auth_pubkey_ok_ed25519
hydra$
So there we have it. A core dump without a binary ?
Also this may be related --->
https://lists.haxx.se/pipermail/libssh2-devel/2023-June/000116.html
that from Paul Howarth :
> Is there anything I could try to make the sshd tests work?
It might be that the builder's directory permissions are a little too
relaxed for sshd's liking. In Fedora I made this change to solve a
similar-looking issue:
--- tests/test_sshd.test
+++ tests/test_sshd.test
@@ -71,6 +71,7 @@ chmod go-rwx \
# shellcheck disable=SC2086
"${SSHD}" \
-f "${SSHD_FIXTURE_CONFIG:-${d}/openssh_server/sshd_config}" \
+ -o 'StrictModes no' \
-o 'Port 4711' \
-h "${d}/openssh_server/ssh_host_rsa_key" \
-h "${d}/openssh_server/ssh_host_ecdsa_key" \
Any thoughts at all would be greatly appreciated.
--
Dennis Clarke
RISC-V/SPARC/PPC/ARM/CISC
just muttering along here
Nope .. that didn't help :
hydra$
hydra$ diff -u ./tests/test_sshd.test.orig ./tests/test_sshd.test
--- ./tests/test_sshd.test.orig 2023-05-30 15:58:13.000000000 +0000
+++ ./tests/test_sshd.test 2023-07-12 09:35:05.256655000 +0000
@@ -71,6 +71,7 @@
# shellcheck disable=SC2086
"${SSHD}" \
-f "${SSHD_FIXTURE_CONFIG:-${d}/openssh_server/sshd_config}" \
+ -o 'StrictModes no' \
-o 'Port 4711' \
-h "${d}/openssh_server/ssh_host_rsa_key" \
-h "${d}/openssh_server/ssh_host_ecdsa_key" \
hydra$
gmake[4]: Entering directory '/opt/bw/build/libssh2-1.11.0_FreeBSD14_amd64.002/tests'
PASS: mansyntax.sh
PASS: test_simple
FAIL: test_sshd.test 1 - sshd-test_ssh2
FAIL: test_sshd.test 2 - sshd-test_auth_pubkey_ok_ed25519
============================================================================
Testsuite summary for libssh2 -
============================================================================
# TOTAL: 4
# PASS: 2
# SKIP: 0
# XFAIL: 0
# FAIL: 2
# XPASS: 0
# ERROR: 0
============================================================================
See tests/test-suite.log
Please report to libssh2-devel@lists.haxx.se
============================================================================
gmake[4]: *** [Makefile:1241: test-suite.log] Error 1
gmake[4]: Leaving directory '/opt/bw/build/libssh2-1.11.0_FreeBSD14_amd64.002/tests'
gmake[3]: *** [Makefile:1349: check-TESTS] Error 2
gmake[3]: Leaving directory '/opt/bw/build/libssh2-1.11.0_FreeBSD14_amd64.002/tests'
gmake[2]: *** [Makefile:1608: check-am] Error 2
gmake[2]: Leaving directory '/opt/bw/build/libssh2-1.11.0_FreeBSD14_amd64.002/tests'
gmake[1]: *** [Makefile:1133: check-recursive] Error 1
gmake[1]: Leaving directory '/opt/bw/build/libssh2-1.11.0_FreeBSD14_amd64.002/tests'
gmake: *** [Makefile:530: check-recursive] Error 1
Also, again, a core file with no binary to be found
hydra$
hydra$ find . -type f | grep 'core'
./tests/test_auth_pubkey_ok.core
hydra$
hydra$ file ./tests/test_auth_pubkey_ok.core
./tests/test_auth_pubkey_ok.core: ELF 64-bit LSB core file, x86-64, version 1 (FreeBSD), FreeBSD-style, from '/opt/bw/build/libssh2-1.11.0_FreeBSD14_amd64.002/tests/.libs/test_auth_pubkey_ok', pid=72030
hydra$
No such binary . What is going on here ?
Yeah, we don't have a test called test_auth_pubkey_ok. The sshd-based tests run two binaries, one of them is test_auth_pubkey_ok_ed25519. test_ssh2 is the other.
You may want to try with the latest master, as it tries to show a bit more info when failing with these tests. Though a core dump for a non-existing binary is indeed misterious.
I've since added FreeBSD 13.2 tests to CI. It's running without issues (aside some flakiness of the runner env) with the same 4 tests: E.g. https://github.com/libssh2/libssh2/actions/runs/7116237839/job/19379170044
Once 14 is available, we can bump and see. Meanwhile maybe you can spot the differences compared to a 14 run.
Added a FreeBSD 14 CI job in 46333ad #1277.
It needed installing bash, then had to add StrictModes no to make it work: https://github.com/libssh2/libssh2/actions/runs/7136414882/job/19434832820
I left more details and log links in the commit.
It didn't replicate the segmentation fault nor a weirdly named binary.
I'm considering this fixed with it working in our CI.
Please feel free to discuss, or reopen if anything still needs fixing here.