xdp_dispatcher fails to load

Question

xdp_dispatcher fails to load

juerlink opened this issue a year ago · comments

Hello,
I work with AF_XDP on an own Ethernet IP-Core in an fpga, where I have written the driver. Everything worked fine on Linux kernel v5.10 using libbpf. Now I migrated to kernel v6.1 and libxdp. When using my own AF_XDP test application I have the problem, that the xdp_dispatcher fails to load. But that happens also, when I use the xdp-tutorial example af_xdp_user.c on an intel i225 device. Btw. the driver itself (send and receive telegrams using AF_XDP) works as expected, because the fallback without dispatcher jumps in.
As a workaround I set the environment variable LIBXDP_SKIP_DISPATCHER to 1. Then the dispatcher is not loaded and I don't get the error message, but then the function xsk_socket__delete( ) fails with this error message:
libxdp: No XDP dispatcher found on ifindex 8
It seems xsk_socket__delete( ) tries to unload the dispatcher which was not loaded because of the fallback mode.

This is the output, when I leave LIBXDP_SKIP_DISPATCHER at 0 (try to use the dispatcher):

libbpf: elf: skipping unrecognized data section(8) .xdp_run_config
libbpf: elf: skipping unrecognized data section(9) xdp_metadata
libbpf: elf: skipping unrecognized data section(7) xdp_metadata
libbpf: elf: skipping unrecognized data section(7) xdp_metadata
libbpf: prog 'xdp_dispatcher': BPF program load failed: Invalid argument
libbpf: prog 'xdp_dispatcher': -- BEGIN PROG LOAD LOG --
Func#11 is safe for any args that match its prototype
btf_vmlinux is malformed
reg type unsupported for arg#0 function xdp_dispatcher#29
0: R1=ctx(off=0,imm=0) R10=fp0
; int xdp_dispatcher(struct xdp_md *ctx)
0: (bf) r6 = r1 ; R1=ctx(off=0,imm=0) R6_w=ctx(off=0,imm=0)
1: (b7) r0 = 2 ; R0_w=P2
; __u8 num_progs_enabled = conf.num_progs_enabled;
2: (18) r8 = 0xffff90455b805950 ; R8_w=map_value(off=0,ks=4,vs=124,imm=0)
4: (71) r7 = *(u8 *)(r8 +2) ; R7=P1 R8=map_value(off=0,ks=4,vs=124,imm=0)
; if (num_progs_enabled < 1)
5: (15) if r7 == 0x0 goto pc+141 ; R7=P1
; ret = prog0(ctx);
6: (bf) r1 = r6 ; R1_w=ctx(off=0,imm=0) R6=ctx(off=0,imm=0)
7: (85) call pc+140
btf_vmlinux is malformed
R1 type=ctx expected=fp
Caller passes invalid args into func#1
processed 84 insns (limit 1000000) max_states_per_insn 0 total_states 9 peak_states 9 mark_read 1
-- END PROG LOAD LOG --
libbpf: prog 'xdp_dispatcher': failed to load: -22
libbpf: failed to load object './xdp-dispatcher.o'
libxdp: Failed to load dispatcher: Invalid argument
libxdp: Falling back to loading single prog without dispatcher
libbpf: elf: skipping unrecognized data section(7) xdp_metadata

Do you have any ideas, what I am doing wrong? Should the dispatcher also work on kernel 6.1?
Our hardware is an embedded system using an Intel Core I7 TigerLake platform. We use kernel v6.1 on an Ubuntu Core 22 system with self build kernel. The Kernel is based on an Intel kernel branch:
https://github.com/intel/linux-intel-lts/releases/tag/lts-v6.1.33-rt11-preempt-rt-230625T232749Z

Thanks a lot
Juergen

Toke Høiland-Jørgensen · Answer 1 · Mon Aug 07 2023 23:52:19 GMT+0800 (China Standard Time)

juerlink ***@***.***> writes:

Hello, I work with AF_XDP on an own Ethernet IP-Core in an fpga, where I have written the driver. Everything worked fine on Linux kernel v5.10 using libbpf. Now I migrated to kernel v6.1 and libxdp. When using my own AF_XDP test application I have the problem, that the xdp_dispatcher fails to load. But that happens also, when I use the xdp-tutorial example af_xdp_user.c on an intel i225 device. Btw. the driver itself (send and receive telegrams using AF_XDP) works as expected, because the fallback without dispatcher jumps in. As a workaround I set the environment variable LIBXDP_SKIP_DISPATCHER to 1. Then the dispatcher is not loaded and I don't get the error message, but then the function xsk_socket__delete( ) fails with this error message: libxdp: No XDP dispatcher found on ifindex 8 It seems xsk_socket__delete( ) tries to unload the dispatcher which was not loaded because of the fallback mode. This is the output, when I leave LIBXDP_SKIP_DISPATCHER at 0 (try to use the dispatcher): libbpf: elf: skipping unrecognized data section(8) .xdp_run_config libbpf: elf: skipping unrecognized data section(9) xdp_metadata libbpf: elf: skipping unrecognized data section(7) xdp_metadata libbpf: elf: skipping unrecognized data section(7) xdp_metadata libbpf: prog 'xdp_dispatcher': BPF program load failed: Invalid argument libbpf: prog 'xdp_dispatcher': -- BEGIN PROG LOAD LOG -- Func#11 is safe for any args that match its prototype btf_vmlinux is malformed

^^^ I think this is your issue. The dispatcher requires that BTF information is available in the running kernel, which does not seem to be the case here (or there's some bug in it). Most likely this is an issue of how you're compiling your kernel. Make sure you have enabled BTF in the kernel build (/sys/kernel/btf/vmlinux should exist and you should be able to dump it with `bpftool btf dump file /sys/kernel/btf/vmlinux`).

juerlink · Answer 2 · Tue Aug 08 2023 01:40:46 GMT+0800 (China Standard Time)

Thanks a lot, I will check that

Sachin Tiptur · Answer 3 · Thu Sep 07 2023 18:36:22 GMT+0800 (China Standard Time)

I have a similar issue while loading basic01-xdp-pass example program from xdp-tutorial. Though i am on newer kernel 5.15 but in my case its arm64. Is there any restriction w.r.t arm64/aarch64 ??

Toke Høiland-Jørgensen · Answer 4 · Thu Sep 07 2023 18:49:42 GMT+0800 (China Standard Time)

Sachin Tiptur ***@***.***> writes:

I have a similar issue while loading `basic01-xdp-pass` example program from xdp-tutorial. Though i am on newer kernel 5.15 but in my case its arm64. Is there any restriction w.r.t `arm64/aarch64` ??

Yes, for Arm you'll need a newer kernel - the dispatcher relies on a JIT feature which is not available on ARM in 5.15...

juerlink · Answer 5 · Thu Sep 07 2023 19:10:24 GMT+0800 (China Standard Time)

^^^ I think this is your issue. The dispatcher requires that BTF information is available in the running kernel, which does not seem to be the case here (or there's some bug in it).

We could build our kernel (X86 V6.1) with BTF included, but the kernel snap got really big, too big for our embedded system (1,4 GBytes - 360 MBytes before). Actually our application also works with libbpf from kernel 5.10 (our previously used version). So for the moment we stay with that libbpf. As next step we try to use libxdp with LIBXDP_SKIP_DISPATCHER == 1 and try to find out, why libxdp tries to unload the dispatcher even if it never was loaded. This seems to be a bug in libxdp.

Toke Høiland-Jørgensen · Answer 6 · Thu Sep 07 2023 19:17:56 GMT+0800 (China Standard Time)

juerlink ***@***.***> writes:

> ^^^ I think this is your issue. The dispatcher requires that BTF information is available in the running kernel, which does not seem to be the case here (or there's some bug in it). We could build our kernel (X86 V6.1) with BTF included, but the kernel snap got really big, too big for our embedded system (1,4 GBytes - 360 MBytes before).

That sounds like you're keeping the entire DWARF debug information. That's not needed, only the BTF information, which is a few dozen MB or so at most.

Actually our application also works with libbpf from kernel 5.10 (our previously used version). So for the moment we stay with that libbpf. As next step we try to use libxdp with LIBXDP_SKIP_DISPATCHER == 1 and try to find out, why libxdp tries to unload the dispatcher even if it never was loaded. This seems to be a bug in libxdp.

Could you describe this issue in more detail, please? Probably better to open a separate Github issue for this, actually...

juerlink · Answer 7 · Thu Sep 07 2023 19:42:27 GMT+0800 (China Standard Time)

That sounds like you're keeping the entire DWARF debug information. That's not needed, only the BTF information, which is a few dozen MB or so at most.

Thanks a lot. Yes that may be the case. I'm not so familiar with DWARF. A colleague helped me. I will give it a new try.

Concerning the other issue:
I mentioned that already in the first post:

As a workaround I set the environment variable LIBXDP_SKIP_DISPATCHER to 1. Then the dispatcher is not loaded and I don't get the error message, but then the function xsk_socket__delete( ) fails with this error message:
libxdp: No XDP dispatcher found on ifindex 8

When the user application is finished, it calls xsk_socket__delete() and this brings the above error.

juerlink · Answer 8 · Fri Oct 06 2023 12:41:37 GMT+0800 (China Standard Time)

We decided to try it first without the dispatcher, because in our special case we want to use AF_XDP for only one channel. It's an industrial real time application and our network driver and hardware use AF_XDP with zero copy mainly for getting low latency.
Thanks to all, I close the issue