xdp-trafficgen not transmitting packets - ndo_xdp_xmit resulting in error -6 on virtio net
tariromukute opened this issue · comments
Hi,
I'm working on running xdp-trafficgen within an OpenStack VM and encountering an error when sending packets.
VM Configuration:
Virtio-net with multiqueue enabled
4 vCPUs
4 combined queues (output of ethtool -l enp4s0)
xdp-trafficgen Output:
lo->enp4s0 327,680 err/s 0 xmit/s
lo->enp4s0 328,332 err/s 0 xmit/s
Tracing Output:
Enabling tracing (echo 1 > /sys/kernel/debug/tracing/events/xdp/enable ; cat /sys/kernel/debug/tracing/trace_pipe) indicates that ndo_xdp_xmit results in error -6
xdp-trafficgen-6322 [000] b...1 1019.407753: xdp_redirect: prog_id=123 action=REDIRECT ifindex=1 to_ifindex=3 err=0 map_id=2147483647 map_index=0
xdp-trafficgen-6322 [000] b...1 1019.407753: xdp_redirect: prog_id=123 action=REDIRECT ifindex=1 to_ifindex=3 err=0 map_id=2147483647 map_index=0
xdp-trafficgen-6322 [000] b...1 1019.407753: xdp_devmap_xmit: ndo_xdp_xmit from_ifindex=1 to_ifindex=3 action=REDIRECT sent=0 drops=16 err=-6
Any insights on how to resolve this issue? Any configuration pointers or debugging suggestions for XDP within OpenStack would be greatly appreciated.
tariromukute ***@***.***> writes:
Hi,
I'm working on running xdp-trafficgen within an OpenStack VM and encountering an error when sending packets.
**VM Configuration**:
Virtio-net with multiqueue enabled
4 vCPUs
4 combined queues (output of ethtool -l enp4s0)
**xdp-trafficgen Output**:
```
lo->enp4s0 327,680 err/s 0 xmit/s
lo->enp4s0 328,332 err/s 0 xmit/s
```
**Tracing Output**:
Enabling tracing (`echo 1 > /sys/kernel/debug/tracing/events/xdp/enable ; cat /sys/kernel/debug/tracing/trace_pipe`) indicates that ndo_xdp_xmit results in error -6
```
xdp-trafficgen-6322 [000] b...1 1019.407753: xdp_redirect: prog_id=123 action=REDIRECT ifindex=1 to_ifindex=3 err=0 map_id=2147483647 map_index=0
xdp-trafficgen-6322 [000] b...1 1019.407753: xdp_redirect: prog_id=123 action=REDIRECT ifindex=1 to_ifindex=3 err=0 map_id=2147483647 map_index=0
xdp-trafficgen-6322 [000] b...1 1019.407753: xdp_devmap_xmit: ndo_xdp_xmit from_ifindex=1 to_ifindex=3 action=REDIRECT sent=0 drops=16 err=-6
```
Any insights on how to resolve this issue? Any configuration pointers
or debugging suggestions for XDP within OpenStack would be greatly
appreciated.
virtio_net can only be a redirect target if an XDP program is loaded on
the interface; so you'll have to load a dummy program on the interface
before using xdp-trafficgen...
Hi Toke,
Thanks for responding to my query. I am not the openstack admin. I only
have access to the VM. Does this mean I won’t be able to run the
xdp-trafficgen on the VM?
If I set the port type to macvlan or direct-physical, will I be able to run
xdp-trafficgen?
P.S: I don’t have access to those port types at the moment, but I assume if
that’s the solution, the administrators will elevate my access/capabilities.
Thanks,
Tariro
…On Wed, 06 Mar 2024 at 13:15, Toke Høiland-Jørgensen < ***@***.***> wrote:
tariromukute ***@***.***> writes:
> Hi,
>
> I'm working on running xdp-trafficgen within an OpenStack VM and
encountering an error when sending packets.
>
> **VM Configuration**:
> Virtio-net with multiqueue enabled
> 4 vCPUs
> 4 combined queues (output of ethtool -l enp4s0)
>
> **xdp-trafficgen Output**:
> ```
> lo->enp4s0 327,680 err/s 0 xmit/s
> lo->enp4s0 328,332 err/s 0 xmit/s
> ```
>
> **Tracing Output**:
> Enabling tracing (`echo 1 > /sys/kernel/debug/tracing/events/xdp/enable
; cat /sys/kernel/debug/tracing/trace_pipe`) indicates that ndo_xdp_xmit
results in error -6
>
> ```
> xdp-trafficgen-6322 [000] b...1 1019.407753: xdp_redirect: prog_id=123
action=REDIRECT ifindex=1 to_ifindex=3 err=0 map_id=2147483647 map_index=0
> xdp-trafficgen-6322 [000] b...1 1019.407753: xdp_redirect: prog_id=123
action=REDIRECT ifindex=1 to_ifindex=3 err=0 map_id=2147483647 map_index=0
> xdp-trafficgen-6322 [000] b...1 1019.407753: xdp_devmap_xmit:
ndo_xdp_xmit from_ifindex=1 to_ifindex=3 action=REDIRECT sent=0 drops=16
err=-6
> ```
>
> Any insights on how to resolve this issue? Any configuration pointers
> or debugging suggestions for XDP within OpenStack would be greatly
> appreciated.
virtio_net can only be a redirect target if an XDP program is loaded on
the interface; so you'll have to load a dummy program on the interface
before using xdp-trafficgen...
—
Reply to this email directly, view it on GitHub
<#400 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AENIPVQQSJS2L2ETYREJD5DYW33FJAVCNFSM6AAAAABEGJCBBOVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTSOBQGYZTQMJTG4>
.
You are receiving this because you authored the thread.Message ID:
***@***.***>
tariromukute ***@***.***> writes:
Hi Toke,
Thanks for responding to my query. I am not the openstack admin. I only
have access to the VM. Does this mean I won’t be able to run the
xdp-trafficgen on the VM?
Probably? Not entirely sure, I don't run many VMs myself.
If I set the port type to macvlan or direct-physical, will I be able to run
xdp-trafficgen?
macvlan probably not, direct-physical, I dunno, what does that do? If it
just passes the physical device through from the host, whether that will
work will depend on whether there's support for XDP in the driver of
whatever physical NIC is on the host...
Thanks for the details. I saw this blog that stated that they got it to work by using macvlan bridge.
the physical device through from the host, whether that will
work will depend on whether there's support for XDP in the driver of
whatever physical NIC is on the host
This makes sense, I will confirm which driver is being used on the compute nodes and see if it will work. Then give an update here.
Thanks.
tariromukute ***@***.***> writes:
Thanks for the details. I saw [this
blog](https://fedepaol.github.io/blog/2023/09/11/xdp-ate-my-packets-and-how-i-debugged-it/)
that stated that they got it to work by using macvlan bridge.
That seems odd. The macvlan device type itself definitely doesn't have
any XDP support. Dunno how Docker sets that up, though, there may be
some passthrough thing doing on where it's really the underlying device
that gets a program attached? No idea, really...
Hi @tohojo, I haven't been able to try the solutions you suggested above on the VMs. We are opting to get a network device that would support BPF_F_TEST_XDP_LIVE_FRAMES and hence the xdp-trafficgen. Tried with I210 NICs with igb driver and got the same error. To ensure compatibility, could you please recommend network devices that work with BPF_F_TEST_XDP_LIVE_FRAMES and xdp-trafficgen. I am aware of the support list here, however it was last updated 5 years ago. Thanks.
tariromukute ***@***.***> writes:
Hi @tohojo, I haven't been able to try the solutions you suggested above on the VMs. We are opting to get a network device that would support `BPF_F_TEST_XDP_LIVE_FRAMES` and hence the `xdp-trafficgen`. Tried with I210 NICs with igb driver and got the same error. To ensure compatibility, could you please recommend network devices that work with `BPF_F_TEST_XDP_LIVE_FRAMES` and `xdp-trafficgen`. Thanks.
There's a table of driver support that's relatively up to date in my
NetConf presentation from the fall, here: https://github.com/xdp-project/xdp-project/blob/master/conference/NetConf2023/xdp-past-present-future.pdf
Thanks @tohojo for the updated list! I managed to get it working with
hv_netvsc on an Azure VM with accelerated networking. I didn’t face any
challenges in getting it to run. However, the drawback was the Azure flow
rules that drop some packets to avoid flooding their network I assume.
On the drivers that have configuration-dependent, is there anywhere I can
find the required configuration for a specific driver. We are looking at
getting the mlx5 compatible devices, any pointers?
…On Tue, 02 Apr 2024 at 23:33, Toke Høiland-Jørgensen < ***@***.***> wrote:
tariromukute ***@***.***> writes:
> Hi @tohojo, I haven't been able to try the solutions you suggested above
on the VMs. We are opting to get a network device that would support
`BPF_F_TEST_XDP_LIVE_FRAMES` and hence the `xdp-trafficgen`. Tried with
I210 NICs with igb driver and got the same error. To ensure compatibility,
could you please recommend network devices that work with
`BPF_F_TEST_XDP_LIVE_FRAMES` and `xdp-trafficgen`. Thanks.
There's a table of driver support that's relatively up to date in my
NetConf presentation from the fall, here:
https://github.com/xdp-project/xdp-project/blob/master/conference/NetConf2023/xdp-past-present-future.pdf
—
Reply to this email directly, view it on GitHub
<#400 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AENIPVRRTQN366MOJHZ6UL3Y3MP2NAVCNFSM6AAAAABEGJCBBOVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDAMZTGEZTMOBYGI>
.
You are receiving this because you authored the thread.Message ID:
***@***.***>
tariromukute ***@***.***> writes:
Thanks @tohojo for the updated list! I managed to get it working with
hv_netvsc on an Azure VM with accelerated networking. I didn’t face any
challenges in getting it to run. However, the drawback was the Azure flow
rules that drop some packets to avoid flooding their network I assume.
On the drivers that have configuration-dependent, is there anywhere I can
find the required configuration for a specific driver. We are looking at
getting the mlx5 compatible devices, any pointers?
I don't have a reference for that handy, no. Whenever I have used mlx5
cards myself, XDP has been working in the default configuration, but
this may well be hardware-dependent, so YMMV...