ERROR: Can't load eBPF object: Argument list too long(-7)

Question

ERROR: Can't load eBPF object: Argument list too long(-7)

shuiouyfm opened this issue 2 years ago · comments

hi,

I load the xdp file with ./xdp-loader load veth9ebc34d af_xdp_kern.o
sudo ./xdpdump -D shows

Interface Prio Program name Mode ID Tag Chain actions

lo <No XDP program loaded!>
ens1f0 <No XDP program loaded!>
enp0s31f6 <No XDP program loaded!>
veth9ebc34d xdp_sock_prog native 1941 84dbbb323171de50
veth929ee17 <No XDP program loaded!>

when I run cmd :sudo ./xdpdump -i veth9ebc34d -p xdp_sock_prog -v -x
it tips:
Current rlimit 67108864 already >= minimum 1048576
Found func xdp_sock_prog matching xdp_sock_prog
Looking for './xdpdump_bpf.o'
Loading bpf file 'xdpdump_bpf.o' from './xdpdump_bpf.o'
ERROR: Can't load eBPF object: Argument list too long(-7)
the kernel version info：
Linux buildserver 5.4.0-104-generic #118-Ubuntu SMP Wed Mar 2 19:02:41 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux

Do I make a mistake of it ? how can I do next step?

Eelco Chaudron · Answer 1 · Fri Mar 18 2022 17:33:55 GMT+0800 (China Standard Time)

Looks like your kernel does not support multi-program loading.
Do you get the same error if you do not supply the -p option?
Can you dump the full output of -vv if this also does not work, i.e. :

xdpdump -i veth9ebc34d -vv

shuiouyfm · Answer 2 · Fri Mar 18 2022 18:03:39 GMT+0800 (China Standard Time)

hi, thanks your reply. the output as follows:
[feyuan@10.224.170.142 ~/xdp-tools/xdp-dump]$sudo ./xdpdump -i veth9ebc34d -vv
Current rlimit 67108864 already >= minimum 1048576
libxdp: DATASEC 'xdp_metadata' not found.
libxdp: DATASEC '.xdp_run_config' not found.
libxdp: Found legacy program with id 1941 and 0 component progs
Found func xdp_sock_prog matching xdp_sock_prog
Looking for './xdpdump_bpf.o'
Loading bpf file 'xdpdump_bpf.o' from './xdpdump_bpf.o'
libbpf: loading ./xdpdump_bpf.o
libbpf: elf: section(3) fentry/func, size 304, link 0, flags 6, type=1
libbpf: sec 'fentry/func': found program 'trace_on_entry' at insn offset 0 (0 bytes), code size 38 insns (304 bytes)
libbpf: elf: section(4) .relfentry/func, size 32, link 26, flags 0, type=9
libbpf: elf: section(5) fexit/func, size 320, link 0, flags 6, type=1
libbpf: sec 'fexit/func': found program 'trace_on_exit' at insn offset 0 (0 bytes), code size 40 insns (320 bytes)
libbpf: elf: section(6) .relfexit/func, size 32, link 26, flags 0, type=9
libbpf: elf: section(7) license, size 4, link 0, flags 3, type=1
libbpf: license of ./xdpdump_bpf.o is GPL
libbpf: elf: section(8) .maps, size 32, link 0, flags 3, type=1
libbpf: elf: section(9) .data, size 12, link 0, flags 3, type=1
libbpf: elf: section(18) .BTF, size 1855, link 0, flags 0, type=1
libbpf: elf: section(20) .BTF.ext, size 1132, link 0, flags 0, type=1
libbpf: elf: section(26) .symtab, size 1848, link 1, flags 0, type=2
libbpf: looking for externs among 77 symbols...
libbpf: collected 0 externs total
libbpf: map 'xdpdump_perf_map': at sec_idx 8, offset 0.
libbpf: map 'xdpdump_perf_map': found type = 4.
libbpf: map 'xdpdump_perf_map': found key [3], sz = 4.
libbpf: map 'xdpdump_perf_map': found value [10], sz = 4.
libbpf: map 'xdpdump_perf_map': found max_entries = 256.
libbpf: map 'xdpdump_.data' (global data): at sec_idx 9, offset 0, flags 400.
libbpf: map 1 is "xdpdump_.data"
libbpf: sec '.relfentry/func': collecting relocation for section(3) 'fentry/func'
libbpf: sec '.relfentry/func': relo #0: insn #7 against 'trace_cfg'
libbpf: prog 'trace_on_entry': found data map 1 (xdpdump_.data, sec 9, off 0) for insn 7
libbpf: sec '.relfentry/func': relo #1: insn #32 against 'xdpdump_perf_map'
libbpf: prog 'trace_on_entry': found map 0 (xdpdump_perf_map, sec 8, off 0) for insn #32
libbpf: sec '.relfexit/func': collecting relocation for section(5) 'fexit/func'
libbpf: sec '.relfexit/func': relo #0: insn #8 against 'trace_cfg'
libbpf: prog 'trace_on_exit': found data map 1 (xdpdump_.data, sec 9, off 0) for insn 8
libbpf: sec '.relfexit/func': relo #1: insn #34 against 'xdpdump_perf_map'
libbpf: prog 'trace_on_exit': found map 0 (xdpdump_perf_map, sec 8, off 0) for insn #34
libbpf: loading kernel BTF '/sys/kernel/btf/vmlinux': 0
libbpf: map 'xdpdump_perf_map': created successfully, fd=5
libbpf: map 'xdpdump_.data': created successfully, fd=6
libbpf: sec 'fentry/func': found 6 CO-RE relocations
libbpf: CO-RE relocating [17] struct xdp_buff: found target candidate [27985] struct xdp_buff in [vmlinux]
libbpf: prog 'trace_on_entry': relo #0: kind <byte_off> (0), spec is [17] struct xdp_buff.data (0:0 @ offset 0)
libbpf: prog 'trace_on_entry': relo #0: matching candidate #0 [27985] struct xdp_buff.data (0:0 @ offset 0)
libbpf: prog 'trace_on_entry': relo #0: patched insn #1 (LDX/ST/STX) off 0 -> 0
libbpf: prog 'trace_on_entry': relo #1: kind <byte_off> (0), spec is [17] struct xdp_buff.data_end (0:1 @ offset 8)
libbpf: prog 'trace_on_entry': relo #1: matching candidate #0 [27985] struct xdp_buff.data_end (0:1 @ offset 8)
libbpf: prog 'trace_on_entry': relo #1: patched insn #2 (LDX/ST/STX) off 8 -> 8
libbpf: prog 'trace_on_entry': relo #2: kind <byte_off> (0), spec is [17] struct xdp_buff.rxq (0:5 @ offset 40)
libbpf: prog 'trace_on_entry': relo #2: matching candidate #0 [27985] struct xdp_buff.rxq (0:5 @ offset 40)
libbpf: prog 'trace_on_entry': relo #2: patched insn #4 (LDX/ST/STX) off 40 -> 40
libbpf: CO-RE relocating [21] struct xdp_rxq_info: found target candidate [3946] struct xdp_rxq_info in [vmlinux]
libbpf: CO-RE relocating [21] struct xdp_rxq_info: found target candidate [29449] struct xdp_rxq_info in [vmlinux]
libbpf: prog 'trace_on_entry': relo #3: kind <byte_off> (0), spec is [21] struct xdp_rxq_info.dev (0:0 @ offset 0)
libbpf: prog 'trace_on_entry': relo #3: matching candidate #0 [3946] struct xdp_rxq_info.dev (0:0 @ offset 0)
libbpf: prog 'trace_on_entry': relo #3: matching candidate #1 [29449] struct xdp_rxq_info.dev (0:0 @ offset 0)
libbpf: prog 'trace_on_entry': relo #3: patched insn #5 (LDX/ST/STX) off 0 -> 0
libbpf: CO-RE relocating [23] struct net_device: found target candidate [3215] struct net_device in [vmlinux]
libbpf: CO-RE relocating [23] struct net_device: found target candidate [29088] struct net_device in [vmlinux]
libbpf: prog 'trace_on_entry': relo #4: kind <byte_off> (0), spec is [23] struct net_device.ifindex (0:0 @ offset 0)
libbpf: prog 'trace_on_entry': relo #4: matching candidate #0 [3215] struct net_device.ifindex (0:22 @ offset 264)
libbpf: prog 'trace_on_entry': relo #4: matching candidate #1 [29088] struct net_device.ifindex (0:22 @ offset 264)
libbpf: prog 'trace_on_entry': relo #4: patched insn #6 (LDX/ST/STX) off 0 -> 264
libbpf: prog 'trace_on_entry': relo #5: kind <byte_off> (0), spec is [21] struct xdp_rxq_info.queue_index (0:1 @ offset 8)
libbpf: prog 'trace_on_entry': relo #5: matching candidate #0 [3946] struct xdp_rxq_info.queue_index (0:1 @ offset 8)
libbpf: prog 'trace_on_entry': relo #5: matching candidate #1 [29449] struct xdp_rxq_info.queue_index (0:1 @ offset 8)
libbpf: prog 'trace_on_entry': relo #5: patched insn #15 (LDX/ST/STX) off 8 -> 8
libbpf: sec 'fexit/func': found 6 CO-RE relocations
libbpf: prog 'trace_on_exit': relo #0: kind <byte_off> (0), spec is [17] struct xdp_buff.data (0:0 @ offset 0)
libbpf: prog 'trace_on_exit': relo #0: matching candidate #0 [27985] struct xdp_buff.data (0:0 @ offset 0)
libbpf: prog 'trace_on_exit': relo #0: patched insn #2 (LDX/ST/STX) off 0 -> 0
libbpf: prog 'trace_on_exit': relo #1: kind <byte_off> (0), spec is [17] struct xdp_buff.data_end (0:1 @ offset 8)
libbpf: prog 'trace_on_exit': relo #1: matching candidate #0 [27985] struct xdp_buff.data_end (0:1 @ offset 8)
libbpf: prog 'trace_on_exit': relo #1: patched insn #3 (LDX/ST/STX) off 8 -> 8
libbpf: prog 'trace_on_exit': relo #2: kind <byte_off> (0), spec is [17] struct xdp_buff.rxq (0:5 @ offset 40)
libbpf: prog 'trace_on_exit': relo #2: matching candidate #0 [27985] struct xdp_buff.rxq (0:5 @ offset 40)
libbpf: prog 'trace_on_exit': relo #2: patched insn #5 (LDX/ST/STX) off 40 -> 40
libbpf: prog 'trace_on_exit': relo #3: kind <byte_off> (0), spec is [21] struct xdp_rxq_info.dev (0:0 @ offset 0)
libbpf: prog 'trace_on_exit': relo #3: matching candidate #0 [3946] struct xdp_rxq_info.dev (0:0 @ offset 0)
libbpf: prog 'trace_on_exit': relo #3: matching candidate #1 [29449] struct xdp_rxq_info.dev (0:0 @ offset 0)
libbpf: prog 'trace_on_exit': relo #3: patched insn #6 (LDX/ST/STX) off 0 -> 0
libbpf: prog 'trace_on_exit': relo #4: kind <byte_off> (0), spec is [23] struct net_device.ifindex (0:0 @ offset 0)
libbpf: prog 'trace_on_exit': relo #4: matching candidate #0 [3215] struct net_device.ifindex (0:22 @ offset 264)
libbpf: prog 'trace_on_exit': relo #4: matching candidate #1 [29088] struct net_device.ifindex (0:22 @ offset 264)
libbpf: prog 'trace_on_exit': relo #4: patched insn #7 (LDX/ST/STX) off 0 -> 264
libbpf: prog 'trace_on_exit': relo #5: kind <byte_off> (0), spec is [21] struct xdp_rxq_info.queue_index (0:1 @ offset 8)
libbpf: prog 'trace_on_exit': relo #5: matching candidate #0 [3946] struct xdp_rxq_info.queue_index (0:1 @ offset 8)
libbpf: prog 'trace_on_exit': relo #5: matching candidate #1 [29449] struct xdp_rxq_info.queue_index (0:1 @ offset 8)
libbpf: prog 'trace_on_exit': relo #5: patched insn #17 (LDX/ST/STX) off 8 -> 8
libbpf: prog 'trace_on_entry': BPF program load failed: Argument list too long
libbpf: failed to load program 'trace_on_entry'
libbpf: failed to load object './xdpdump_bpf.o'
ERROR: Can't load eBPF object: Argument list too long(-7)

Eelco Chaudron · Answer 3 · Fri Mar 18 2022 23:43:50 GMT+0800 (China Standard Time)

Not sure what happens here. Quickly browsing the kernel code I could be the program has too many instructions, but I think we can rule that out. Or there is something that goes wrong with copying the data from userspace in libbpf.

@tohojo do you know anything about libbpf compatibility problems?

Toke Høiland-Jørgensen · Answer 4 · Sat Mar 19 2022 00:20:00 GMT+0800 (China Standard Time)

@tohojo do you know anything about libbpf compatibility problems?

Nope :(

Eelco Chaudron · Answer 5 · Tue Mar 22 2022 16:33:36 GMT+0800 (China Standard Time)

@shuiouyfm did you build with the embedded libbpf or with the system one?

If you're lib/libbpf directory has no content, you build with the system included one. To try and build with the embedded one, you can do FORCE_SUBDIR_LIBBPF=1 ./configure and rebuild.

If you do not have the system libbpf library installed, can you try installing those, and do another ./configure, you can remove the ./lib/libbpf submodule to make sure its not re-used.

shuiouyfm · Answer 6 · Wed Mar 23 2022 15:26:00 GMT+0800 (China Standard Time)

@chaudron, as your memtioned , my kernel version is 5.4 and I found some info as :

In other words, kernels before 5.6 can only attach a single XDP program to each interface, kernels 5.6+ can attach multiple programs if they are all attached at the same time, and kernels 5.10 have full support for XDP multiprog on x86_64.
from https://www.mankier.com/3/libxdp#Kernel_and_BPF_program_feature_compatibility

I guess this is the root cause.

And I build with the embedded lib/libbpf.

thank you.

Eelco Chaudron · Answer 7 · Wed Mar 23 2022 20:29:32 GMT+0800 (China Standard Time)

I missed the kernel part :( So for XDP dump the kernel needs to at least have the below three commits. And from the git repo data, you need at least 5.10 also.

$ git tag --contains d831ee84bfc9 | grep -v rc | sort -n
v5.10
v5.11
v5.12
v5.13
v5.14
v5.15
v5.16
v5.7
v5.8
v5.9


$ git tag --contains 5b92a28aae4d | grep -v rc | sort -n
v5.10
v5.11
v5.12
v5.13
v5.14
v5.15
v5.16
v5.5
v5.6
v5.7
v5.8
v5.9

$ git tag --contains 43bc2874e779 | grep -v rc | sort -n
v5.10
v5.11
v5.12
v5.13
v5.14
v5.15
v5.16

So I guess you can close this issue, due to an unsupported kernel version.

Toke Høiland-Jørgensen · Answer 8 · Wed Mar 23 2022 20:35:53 GMT+0800 (China Standard Time)

Eelco Chaudron ***@***.***> writes:

So I guess you can close this issue, due to an unsupported kernel version.

But shouldn't xdpdump print a nice error message about this now? So why isn't it?

Eelco Chaudron · Answer 9 · Wed Mar 23 2022 20:51:16 GMT+0800 (China Standard Time)

Well, the error might not be because of the missing commits, looks more like a compatibility issue with libbpf, but not 100% sure. With the missing patches, I think I've seen different error messages in the past.

Toke Høiland-Jørgensen · Answer 10 · Wed Mar 23 2022 20:56:50 GMT+0800 (China Standard Time)

So should we do something like this?

diff --git a/xdp-dump/xdpdump.c b/xdp-dump/xdpdump.c
index cf201e830a2f..48c4c0cf8771 100644
--- a/xdp-dump/xdpdump.c
+++ b/xdp-dump/xdpdump.c
@@ -1419,7 +1419,7 @@ rlimit_loop:
                trace_link_fentry = bpf_program__attach_trace(trace_prog_fentry);
                err = libbpf_get_error(trace_link_fentry);
                if (err) {
-                       if (err == -ENOTSUPP)
+                       if (err == -ENOTSUPP || err == -E2BIG)
 #if defined(__x86_64__) || defined(__i686__)
                                pr_warn("ERROR: The kernel does not support "
                                        "fentry function attach because it is "

Eelco Chaudron · Answer 11 · Wed Mar 23 2022 22:06:22 GMT+0800 (China Standard Time)

@tohojo I looked a bit at the code again, and I do not see a real problem that could indicate a real problem that reports E2BIG for the load program. So to solve this we need to add this check to the "err = bpf_object__load(trace_obj);" part of the code, not the attach part.

Toke Høiland-Jørgensen · Answer 12 · Wed Mar 23 2022 23:19:36 GMT+0800 (China Standard Time)

@shuiouyfm could you please test the patch in #179 and see if you get a nicer error message with that? :)

shuiouyfm · Answer 13 · Thu Mar 24 2022 13:35:54 GMT+0800 (China Standard Time)

hi
the error message is printed in my case
[feyuan@10.224.170.142 ~/xdp-tools/xdp-dump]$sudo ./xdpdump -i veth9ebc34d -p xdp_sock_prog -v -x
Current rlimit 67108864 already >= minimum 1048576
Found func xdp_sock_prog matching xdp_sock_prog
Looking for './xdpdump_bpf.o'
Loading bpf file 'xdpdump_bpf.o' from './xdpdump_bpf.o'
ERROR: The kernel does not support fentry function load because it is too old!

Toke Høiland-Jørgensen · Answer 14 · Thu Mar 24 2022 17:18:24 GMT+0800 (China Standard Time)

shuiouyfm ***@***.***> writes:

hi the error message is printed in my case ***@***.*** ~/xdp-tools/xdp-dump]$sudo ./xdpdump -i veth9ebc34d -p xdp_sock_prog -v -x Current rlimit 67108864 already >= minimum 1048576 Found func xdp_sock_prog matching xdp_sock_prog Looking for './xdpdump_bpf.o' Loading bpf file 'xdpdump_bpf.o' from './xdpdump_bpf.o' ERROR: The kernel does not support fentry function load because it is too old!

Awesome! Thanks for testing!