Building pmacct 1.7.8 with nDPI 4.6-stable failing

Question

Building pmacct 1.7.8 with nDPI 4.6-stable failing

DavidMcEvoy-ApplianSys opened this issue 9 months ago · comments

DavidMcEvoy-ApplianSys commented 9 months ago

Description

I'm getting the following error when building:

gmake[3]: Entering directory '/tmp/packwork-david/default/pmacct/pmacct/src/ndpi'
  CC       ndpi.lo
ndpi.c: In function ‘pm_ndpi_packet_processing’:
ndpi.c:358:29: error: too few arguments to function ‘ndpi_detection_process_packet’
  358 |   flow->detected_protocol = ndpi_detection_process_packet(workflow->ndpi_struct, ndpi_flow,
      |                             ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~
In file included from /tmp/packwork-david/default/pmacct/nDPI/src/include/ndpi_main.h:35,
                 from ../pmacct.h:102,
                 from ndpi.c:27:
/tmp/packwork-david/default/pmacct/nDPI/src/include/ndpi_api.h:323:17: note: declared here
  323 |   ndpi_protocol ndpi_detection_process_packet(struct ndpi_detection_module_struct *ndpi_struct,
      |                 ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~
gmake[3]: *** [Makefile:424: ndpi.lo] Error 1
gmake[3]: Leaving directory '/tmp/packwork-david/default/pmacct/pmacct/src/ndpi'
gmake[2]: *** [Makefile:1344: all-recursive] Error 1
gmake[2]: Leaving directory '/tmp/packwork-david/default/pmacct/pmacct/src'
gmake[1]: *** [Makefile:697: all] Error 2
gmake[1]: Leaving directory '/tmp/packwork-david/default/pmacct/pmacct/src'
make: *** [Makefile:740: all-recursive] Error 1

I've also found a Gentoo bug report, so it doesn't look like it's a build environment issue.

Version

Downloading source from GitGub for both of these:

pmacct: 1.7.8
nDPI: 4.6-stable

Configuration options are:

./configure \
--prefix=/usr \
--libdir=/usr/lib64 \
--sysconfdir=/etc \
--localstatedir=/var \
--enable-ipv6 \
--enable-plabel \
--enable-nflog \
--enable-ndpi \
--enable-threads \
--enable-static=no \
--enable-bgp-bins=no \
--enable-bmp-bins=no \
--enable-st-bins=no \
--with-ndpi-static-lib=/tmp/packwork-david/default/pmacct/nDPI/src/lib/ \
--enable-shared=yes

Paolo Lucente · Answer 1 · Thu Sep 21 2023 23:24:58 GMT+0800 (China Standard Time)

Hi David ( @DavidMcEvoy-ApplianSys ),

This is expected as 1.7.8 does work against 4.0, 4.2 and 4.4. 1.7.9, which is what you have in master here on GitHub, does compile against 4.6 ( https://github.com/pmacct/pmacct/blob/cf08ee0a927407ed0dcaed93d8f429d1dcc74fde /QUICKSTART#L1252 ). The nDPI API has been breaking backward compatibility pretty much at every recent release.

Paolo

DavidMcEvoy-ApplianSys · Answer 2 · Thu Sep 21 2023 23:45:56 GMT+0800 (China Standard Time)

Hi Paolo,

Thanks for the quick response, I didn't spot that 4.6-stable doesn't work with 1.7.8, as I was just using the release notes, sorry!

I've tried the build again, but using nDPI 4.4-stable and am getting a different error:

/usr/bin/ld: ./.libs/libdaemons.a(libdaemons_la-util.o): in function `version_daemon':
util.c:(.text+0x3ede): undefined reference to `ndpi_revision'
/usr/bin/ld: ./.libs/libdaemons.a(libdaemons_la-print_plugin.o): in function `P_cache_purge':
print_plugin.c:(.text+0x474a): undefined reference to `ndpi_get_proto_name'
/usr/bin/ld: print_plugin.c:(.text+0x476c): undefined reference to `ndpi_get_proto_name'
/usr/bin/ld: print_plugin.c:(.text+0x6189): undefined reference to `ndpi_get_proto_name'
/usr/bin/ld: print_plugin.c:(.text+0x61aa): undefined reference to `ndpi_get_proto_name'
/usr/bin/ld: ./.libs/libdaemons.a(ndpi.o): in function `pm_ndpi_free_flow_info_half':
ndpi.c:(.text+0x103): undefined reference to `ndpi_flow_free'
/usr/bin/ld: ndpi.c:(.text+0x119): undefined reference to `ndpi_free'
/usr/bin/ld: ndpi.c:(.text+0x12f): undefined reference to `ndpi_free'
/usr/bin/ld: ./.libs/libdaemons.a(ndpi.o): in function `pm_ndpi_get_flow_info':
ndpi.c:(.text+0x352): undefined reference to `ndpi_flow_malloc'
/usr/bin/ld: ndpi.c:(.text+0x398): undefined reference to `ndpi_malloc'
/usr/bin/ld: ndpi.c:(.text+0x3b3): undefined reference to `ndpi_malloc'
/usr/bin/ld: ./.libs/libdaemons.a(ndpi.o): in function `pm_ndpi_node_guess_undetected_protocol':
ndpi.c:(.text+0x8f7): undefined reference to `ndpi_guess_undetected_protocol'
/usr/bin/ld: ./.libs/libdaemons.a(ndpi.o): in function `pm_ndpi_packet_processing':
ndpi.c:(.text+0xa65): undefined reference to `ndpi_detection_process_packet'
/usr/bin/ld: ndpi.c:(.text+0xad3): undefined reference to `ndpi_detection_giveup'
/usr/bin/ld: ./.libs/libdaemons.a(ndpi.o): in function `pm_ndpi_idle_flows_cleanup':
ndpi.c:(.text+0xca7): undefined reference to `ndpi_free'
collect2: error: ld returned 1 exit status
gmake[3]: *** [Makefile:854: pmacct] Error 1
gmake[3]: Leaving directory '/tmp/packwork-david/default/pmacct/pmacct/src'
gmake[2]: *** [Makefile:1344: all-recursive] Error 1
gmake[2]: Leaving directory '/tmp/packwork-david/default/pmacct/pmacct/src'
gmake[1]: *** [Makefile:697: all] Error 2
gmake[1]: Leaving directory '/tmp/packwork-david/default/pmacct/pmacct/src'
make: *** [Makefile:740: all-recursive] Error 1

Paolo Lucente · Answer 3 · Mon Sep 25 2023 21:24:44 GMT+0800 (China Standard Time)

This one looks like if nDPI is not installed (correctly) in the system. The linker is unable to locate the nDPI lib apparently. After falling back from nDPI 4.6 to 4.4, did you start clean re-compiling everything, ie. both nDPI & pmacct?

DavidMcEvoy-ApplianSys · Answer 4 · Wed Sep 27 2023 16:35:53 GMT+0800 (China Standard Time)

Hi Paulo @paololucente,

I'm building nDPI separately, then copying the files into the local directory tree.

When building pmacct, these are the variables and configuration options being used:

export NDPI_LIBS=-L${WORK}/nDPI/usr/lib
export NDPI_CFLAGS=-I${WORK}/nDPI/usr/include/ndpi

./autogen.sh
./configure \
  --prefix=/usr \
  --libdir=/usr/lib64 \
  --sysconfdir=/etc \
  --localstatedir=/var \
  --enable-nflog \
  --enable-static=no \
  --enable-bgp-bins=no \
  --enable-bmp-bins=no \
  --enable-st-bins=no \
  --enable-ndpi \
  --with-ndpi-static-lib=${WORK}/nDPI/usr/lib \
  --enable-shared=yes

make $NUMJOBS
make DESTDIR="${WORK}/fs" install-strip

I've tried adjusting both the envars, and the --with-ndpi-static-lib option, but if they're missing or incorrect, the build fails, so it's finding those files.

Just in case there's an issue in the nDPI build, these are the files being copied into the $WORK/nDPI directory :

david@cheddar /tmp/packwork-david/default/pmacct/nDPI
 $ find .
./build_parameters.debug
./usr
./usr/bin
./usr/bin/ndpiReader
./usr/share
./usr/share/ndpi
./usr/share/ndpi/ndpiCustomCategory.txt
./usr/share/ndpi/ndpiProtos.txt
./usr/lib
./usr/lib/libndpi.so
./usr/lib/libndpi.so.4.4.0
./usr/lib/libndpi.so.4
./usr/lib/pkgconfig
./usr/lib/pkgconfig/libndpi.pc
./usr/lib/libndpi.a
./usr/include
./usr/include/ndpi
./usr/include/ndpi/ndpi_api.h
./usr/include/ndpi/ndpi_utils.h
./usr/include/ndpi/ndpi_protocol_ids.h
./usr/include/ndpi/ndpi_protocols.h
./usr/include/ndpi/ndpi_config.h
./usr/include/ndpi/ndpi_define.h
./usr/include/ndpi/ndpi_classify.h
./usr/include/ndpi/ndpi_encryption.h
./usr/include/ndpi/ndpi_unix.h
./usr/include/ndpi/ndpi_includes.h
./usr/include/ndpi/ndpi_patricia_typedefs.h
./usr/include/ndpi/ndpi_main.h
./usr/include/ndpi/ndpi_includes_OpenBSD.h
./usr/include/ndpi/ndpi_win32.h

Paolo Lucente · Answer 5 · Wed Sep 27 2023 23:13:14 GMT+0800 (China Standard Time)

I see, David. Can you please show me also how do you build nDPI? My target is to have all the commands you execute, in the right order, so to be able to reproduce the issue. I think above i have got the pmacct part complete.

DavidMcEvoy-ApplianSys · Answer 6 · Wed Sep 27 2023 23:20:52 GMT+0800 (China Standard Time)

The nDPI build is very strait forwards, we checkout commit 2c5258b (4.4-stable release) then:

./autogen.sh
make $NUMJOBS
make install DESTDIR="${WORK}/fs"

Paolo Lucente · Answer 7 · Thu Sep 28 2023 01:08:51 GMT+0800 (China Standard Time)

Thank you! Apparently there was a little mistake in the doc, NDPI_LIBS should be defined as:

export NDPI_LIBS=-L${WORK}/nDPI/usr/lib -lndpi

That made the trick for me. Let me know if this works for you too.

Paolo

DavidMcEvoy-ApplianSys · Answer 8 · Thu Sep 28 2023 21:56:39 GMT+0800 (China Standard Time)

That's allowed the process to get further along, thank you!

We're getting a new error message now, so progress is being made:

/usr/bin/ld: /tmp/packwork-david/default/pmacct/libnetfilter_log/usr/lib64/libnetfilter_log.so: undefined reference to `nfnl_fill_hdr@NFNETLINK_1.0.1'
/usr/bin/ld: /tmp/packwork-david/default/pmacct/libnetfilter_log/usr/lib64/libnetfilter_log.so: undefined reference to `nfnl_fd@NFNETLINK_1.0.1'
/usr/bin/ld: /tmp/packwork-david/default/pmacct/libnetfilter_log/usr/lib64/libnetfilter_log.so: undefined reference to `nfnl_addattr16@NFNETLINK_1.0.1'
/usr/bin/ld: /tmp/packwork-david/default/pmacct/libnetfilter_log/usr/lib64/libnetfilter_log.so: undefined reference to `nfnl_handle_packet@NFNETLINK_1.0.1'
/usr/bin/ld: /tmp/packwork-david/default/pmacct/libnetfilter_log/usr/lib64/libnetfilter_log.so: undefined reference to `nfnl_query@NFNETLINK_1.0.1'
/usr/bin/ld: /tmp/packwork-david/default/pmacct/libnetfilter_log/usr/lib64/libnetfilter_log.so: undefined reference to `nfnl_open@NFNETLINK_1.0.1'
/usr/bin/ld: /tmp/packwork-david/default/pmacct/libnetfilter_log/usr/lib64/libnetfilter_log.so: undefined reference to `nfnl_addattr_l@NFNETLINK_1.0.1'
/usr/bin/ld: /tmp/packwork-david/default/pmacct/libnetfilter_log/usr/lib64/libnetfilter_log.so: undefined reference to `nfnl_callback_register@NFNETLINK_1.0.1'
/usr/bin/ld: /tmp/packwork-david/default/pmacct/libnetfilter_log/usr/lib64/libnetfilter_log.so: undefined reference to `nfnl_addattr32@NFNETLINK_1.0.1'
/usr/bin/ld: /tmp/packwork-david/default/pmacct/libnetfilter_log/usr/lib64/libnetfilter_log.so: undefined reference to `nfnl_unset_sequence_tracking@NFNETLINK_1.0.1'
/usr/bin/ld: /tmp/packwork-david/default/pmacct/libnetfilter_log/usr/lib64/libnetfilter_log.so: undefined reference to `nfnl_rcvbufsiz@NFNETLINK_1.0.1'
/usr/bin/ld: /tmp/packwork-david/default/pmacct/libnetfilter_log/usr/lib64/libnetfilter_log.so: undefined reference to `nfnl_subsys_open@NFNETLINK_1.0.1'
/usr/bin/ld: /tmp/packwork-david/default/pmacct/libnetfilter_log/usr/lib64/libnetfilter_log.so: undefined reference to `nfnl_close@NFNETLINK_1.0.1'
collect2: error: ld returned 1 exit status
gmake[3]: *** [Makefile:878: uacctd] Error 1
gmake[3]: Leaving directory '/tmp/packwork-david/default/pmacct/pmacct/src'
gmake[2]: *** [Makefile:1344: all-recursive] Error 1
gmake[2]: Leaving directory '/tmp/packwork-david/default/pmacct/pmacct/src'
gmake[1]: *** [Makefile:697: all] Error 2
gmake[1]: Leaving directory '/tmp/packwork-david/default/pmacct/pmacct/src'
make: *** [Makefile:740: all-recursive] Error 1

We were using version 1.0.2 of libnfnetlink, but after downgrading to 1.0.1 we still get the same error.

If your build is working can you let me know what versions of:

libmnl
libnfnetlink
libnetfilter_log

You have, and I'll try matching ours.

Paolo Lucente · Answer 9 · Fri Sep 29 2023 01:40:54 GMT+0800 (China Standard Time)

Great to see progress.

I am running Debian 11.7. libmnl is version 1.0.4. Whereas libnfnetlink and libnetfilter_log are version 1.0.1. With these i see success.

DavidMcEvoy-ApplianSys · Answer 10 · Wed Oct 04 2023 17:32:57 GMT+0800 (China Standard Time)

Hi Paolo,

We're using the same versions of those packages, so it's not that. It's definitely something in our build environment that's causing the problem.

Thanks for your help with this!