[newbee] [libxdp] questions about xsk_def_prog and its XSK map

[TheDiveO]

I've looked at the mailing lists (cloud, hardware hints) of the xdp project, and also here for signs of a mailing list or forum where I can post my newbie newbee questions what I have in trying to grok the libxdp architecture and source code; if there is a better place please kindly point me there (except /dev/null) .

1. Is the "default program" (not the dispatcher) from xks_def_prog.c always been constantly loaded or does it get removed when loading at least one component program? That is, can I as an app developer and libxdp user always rely on the default program being active?
2. Is the XSK map used by the default program pinned in the bpffs? I understand this to be outside the core realm of the libxdp protocol documentation document, but if the default program is considered to be always available then how do multiple libxdp instances coordinate on it and use the same XSK map?

[Toke Høiland-Jørgensen]

TheDiveO ***@***.***> writes:

I've looked at the mailing lists (cloud, hardware hints) of the xdp project, and also here for signs of a mailing list or forum where I can post my ~~newbie~~ newbee questions what I have in trying to grok the libxdp architecture and source code; if there is a better place please kindly point me there (except `/dev/null`) .

For xdp-tools and libxdp-specific questions, this repo is fine; we don't have a mailing list specifically for that. For other things, the ***@***.*** mailing list is probably the best bet.

1. Is the "default program" (not the dispatcher) from xks_def_prog.c always been constantly loaded or does it get removed when loading at least one component program? That is, can I as an app developer and libxdp user always rely on the default program being active?

The default program is only loaded when it's needed (i.e., when at least one application uses AF_XDP). There's a refcounting mechanism to make sure it's removed automatically from an interface when no longer needed.

2. Is the XSK map used by the default program pinned in the bpffs? I understand this to be outside the core realm of the libxdp protocol documentation document, but if the default program is considered to be always available then how do multiple libxdp instances coordinate on it and use the same XSK map?

No, it's not pinned. Libxdp will load it from the kernel using bpf_map_get_fd_by_id() - this is one of the operations that requires CAP_SYS_ADMIN, BTW.

[TheDiveO]

For xdp-tools and libxdp-specific questions, this repo is fine; we don't have a mailing list specifically for that. For other things, the @.*** mailing list is probably the best bet.

The mailing list email address unfortunately got very thoroughly sanitized. 🧹

The default program is only loaded when it's needed (i.e., when at least one application uses AF_XDP). There's a refcounting mechanism to make sure it's removed automatically from an interface when no longer needed.

What does "uses AF_XDP" constitute? A call into one of libxdp's XSK creating functions? Or something else?

2. Is the XSK map used by the default program pinned in the bpffs? I understand this to be outside the core realm of the libxdp protocol documentation document, but if the default program is considered to be always available then how do multiple libxdp instances coordinate on it and use the same XSK map?

No, it's not pinned. Libxdp will load it from the kernel using bpf_map_get_fd_by_id() - this is one of the operations that requires CAP_SYS_ADMIN, BTW.

In trying to understand the life cycle(s) of the involved resources here, do I rather manage to understand by pure chance that the default (component) prog is pinned itself? libxdp then (somehow) identifies a pinned component prog to actually be an already loaded default prog and then next discovers the XSK map from the component prog?

Does this setup also mean that the registered/mapped XSKs become immortal as long as the default comp prog is loaded and pinned, even if the process that once it created took a SIGKILL exit or something similar where it could not unmap the XSK?

As you showed me in the other issue, that could all automatically wind down if using a transient bpffs instance. However, if a lot of completely unrelated processes want to use XSKs for multiple separate queues of the same netdev and there is no virtualization support of such a netdev (just lots of queues) then the transient bpffs instance doesn't help, as crashed and restarted processes eat up the available netdev queues like candy. Is there a way to somehow detect "orphaned" XSKs?

[Toke Høiland-Jørgensen]

TheDiveO ***@***.***> writes:

> For xdp-tools and libxdp-specific questions, this repo is fine; we don't have a mailing list specifically for that. For other things, the ***@***.*** mailing list is probably the best bet. The mailing list email address unfortunately got very thoroughly sanitized. 🧹

Heh. Keep forgetting that the github email gateway does that, it's really annoying. It's xdp-newbies at vger dot kernel dot org.

> The default program is only loaded when it's needed (i.e., when at least one application uses AF_XDP). There's a refcounting mechanism to make sure it's removed automatically from an interface when no longer needed. What does "uses AF_XDP" constitute? A call into one of libxdp's XSK creating functions? Or something else?

AF_XDP is the address family in the kernel used for XSK sockets. So yeah, everything you do with an XSK is using AF_XDP, and that's how we usually refer to it in the kernel, to distinguish it from "regular" uses of XDP, which is custom XDP programs running entirely inside the kernel (i.e., no userspace component). Libxdp started out as a way to just manage kernel XDP programs, and we took over the XSK socket support after it was removed from libbpf.

> > 2. Is the XSK map used by the default program pinned in the bpffs? I understand this to be outside the core realm of the libxdp protocol documentation document, but if the default program is considered to be always available then how do multiple libxdp instances coordinate on it and use the same XSK map? > No, it's not pinned. Libxdp will load it from the kernel using bpf_map_get_fd_by_id() - this is one of the operations that requires CAP_SYS_ADMIN, BTW. In trying to understand the life cycle(s) of the involved resources here, do I rather manage to understand by pure chance that the default (component) prog is pinned itself? libxdp then (somehow) identifies a pinned component prog to actually be an already loaded default prog and then next discovers the XSK map from the component prog?

Yeah, the program is pinned as part of the dispatcher like any other XDP program, but the XSK bit of libxdp just calls into the regular libxdp code to get a list of loaded programs on the interface, then tries to find its default prog. The code to do this is here: https://github.com/xdp-project/xdp-tools/blob/master/lib/libxdp/xsk.c#L684

Does this setup also mean that the registered/mapped XSKs become immortal as long as the default comp prog is loaded and pinned, even if the process that once it created took a SIGKILL exit or something similar where it could not unmap the XSK?

No, the socket will get released in the kernel once the last file descriptor referencing it is closed. This will also remove the XSK from all maps. However, the default program will stay attached (and have a wrong refcnt) if there are applications that exit without calling xdp_socket__delete() first.

[TheDiveO]

No, the socket will get released in the kernel once the last file
descriptor referencing it is closed. This will also remove the XSK from
all maps. However, the default program will stay attached (and have a
wrong refcnt) if there are applications that exit without calling
xdp_socket__delete() first.

Now that's crucial to my understanding: I was assuming until now that adding an XSK (using its fd) does increment the XSK's reference count so that even if the fd is closed, the XSK lives on as it is strongly referenced by the map. So is an XSK map instead weakly referencing XSKs...?

As there are probably many refcnts floating around: which one are we speaking about in this case?

[Toke Høiland-Jørgensen]

TheDiveO ***@***.***> writes:

> No, the socket will get released in the kernel once the last file descriptor referencing it is closed. This will also remove the XSK from all maps. However, the default program will stay attached (and have a wrong refcnt) if there are applications that exit without calling xdp_socket__delete() first. Now that's crucial to my understanding: I was assuming until now that adding an XSK (using its fd) does increment the XSK's reference count so that even if the fd is closed, the XSK lives on as it is strongly referenced by the map. So is an XSK map instead _weakly referencing_ XSKs...?

Yeah, actually the XSK takes a reference to the map, not the other way around... :)

[TheDiveO]

Yeah, actually the XSK takes a reference to the map, not the other way around... :)

Oh, that changes the situation considerably! And that seems to be making resource management much cleaner, as I was wondering how to clean up when the process with the XSK fd goes the way of the Blue Norwegian without properly removing it before from the map. Now that looks very much like it is taken care of. Thank you very much for the explanation!

[Toke Høiland-Jørgensen]

Cool! You're welcome :)