core: Allow external control over provider ordering
shefty opened this issue · comments
Provider ordering is currently hard-coded. Allow the use of an external file to specify ordering criteria. This feature was requested by end-users in order to set the defaults independent from using an environment variable.
Would it be feasible to pass an array of strings resp. provider names to fi_getinfo to express my preference of priority?
For this request, that's not usable. The intent with this request is to allow the system manager, which understands the network configuration, control over the ordering, so that applications do not need to change or modify their environments.
For what you're suggesting, I think it could be achieved by the app calling fi_getinfo multiple times, each time requesting a different provider, and stopping once they find one they want.
would a libfabric.conf in /etc or %PROGRAMDATA%/libfabric be feasible? Is Windows a priority for this feature?
Windows isn't a priority, but if it involves parsing a file, the OS shouldn't matter.
I guess the file could contain an ordered list of filters. The output from fi_getinfo could pass through the filter, moving all filtered options to the front of the list. For example, a simple filter could be:
{
prov_name: net
},
{
prov_name: tcp
}
The fi_getinfo() output would move all net provider options first, followed by tcp provider. All others would follow those.
We could add more filtering and control options. Example:
{
prov_name: net,
ep_type: [FI_EP_MSG, FI_EP_RDM],
filter: first,
},
{
prov_name: tcp,
ep_type: FI_EP_RDM,
filter: last,
},
{
prov_name: udp,
filter: remove
}
Windows isn't a priority, but if it involves parsing a file, the OS shouldn't matter.
it doesn't matter for parsing the file, only locating it - since the requirement is essentially no run-time user interaction, libfabric has to know where to look.
I guess the file could contain an ordered list of filters. The output from fi_getinfo could pass through the filter, moving all filtered options to the front of the list. For example, a simple filter could be:
We should probably distinguish util providers from core providers, too. for example, in your first example, net;rxm would move below tcp.
I'd make the filtering explicit, so, yeah, the second filter in my second example wouldn't actually match anything.
clang-format solves a similar challenge with priorities:
IncludeCategories:
- Regex: '^"(llvm|llvm-c|clang|clang-c)/'
Priority: 2
- Regex: '^((<|")(gtest|gmock|isl|json)/)'
Priority: 3
- Regex: '<[[:alnum:].]+>'
Priority: 4
- Regex: '.*'
Priority: 1
I could name providers and give them priorities. Then you can use -1 to exclude providers.
I suggested the jsmn format because that's what we use for fabtests' ubertest and have parsing code for it. It ends up with an input file that's very easy to read and modify. I was trying to use simple examples above, but I expect the ordering will be more involved and require looking at system settings that are partially outside the scope of libfabric. For example, what NIC is being used, the NIC speed, GPUs that are present, etc. We'll need an easy way to expand the filtering.
I'm considering a use case where a customer may have multiple clusters available, and the selection could differ for each cluster. Part of this comes down to whether the user needs a separate file per cluster or a single file for all clusters. The latter seems less likely to lead to problems.
I did not want to question the format of the file. I just wanted to hint at that numerical priorities per provider could be an option with special cases -1 and max int.
Ah, got it, yes, numbers would be better than 'first' 'last'. I was mostly trying to convey that there would be some that we want to move the front of the list, some toward the back of the list, and others to remove completely. Although I don't know that moving any to the back of the list is really all that useful.
That is probably a philosophical question. Is TCP on your IB cluster never or last resort?
On my cluster? TCP is frequent, but that's because we're trying to develop multiple providers on it. :) In more generic situations, it's application dependent. I was more suggesting that there's little benefit to moving providers explicitly to the end of the list, since we can always move everything else ahead of it, if the default ordering doesn't already do that.
Maybe to make the file exhaustive
default: -1
and to eliminate all non-named providers.
As a simple first step, we could start with: /etc/libfabric.conf or libfabric.cfg. I would have that list environment variables and associated values, maybe 1 per line. We have an FI_PROVIDER variable that can enable/disable providers. We could use that to specify a provider order as well (maybe by adding a new prefix character). For more complex ordering, we will need to define something more, but this could be enough to get us started, and gives us a way to set all environment variables through a file.
#8572 added an external way to disable/enable specific providers. It's not the same as allowing full re-ordering, but is likely sufficient for the intended purpose, and gives the system admin some control over how the system is used without requiring users to set environment variables.