[BUG] salt-call on macos waits for reverse dns for ipv6 link-local addresses

[Ross Vandegrift]

Description
On MacOS, salt-call is waiting ~20s to try to look up reverse dns for every ipv6 link-local addresses:

[TRACE   ] Loading core.fqdns grain
[TRACE   ] 'ip' could not be found in the following search path: '[...]'
[DEBUG   ] Unable to resolve address fe80::b90:a02a:71c5:a34e: [Errno 1] Unknown host
(repeat for other interfaces w/link-local inet6 addr)
...
[DEBUG   ] Elapsed time getting FQDNs: 21.88831114768982 seconds

Setup
salt 3004.1 on macos, installed from homebrew.

• on-prem machine
• VM (Virtualbox, KVM, etc. please specify)
• VM running on a cloud service, please be explicit and add details
• container (Kubernetes, Docker, containerd, etc. please specify)
• or a combination, please be explicit
• jails if it is FreeBSD

Steps to Reproduce the behavior

• brew install salt
• sudo salt-call --local --log-level=debug grains.get core.fqdn

Expected behavior
salt should never bother with reverse dns on link-local addresses (this seems to be the behavior on linux).

Versions Report

salt --versions-report

(Provided by running salt --versions-report. Please also mention any differences in master/minion versions.)

Salt Version:
          Salt: 3004.1

Dependency Versions:
          cffi: 1.14.6
      cherrypy: unknown
      dateutil: 2.8.0
     docker-py: Not Installed
         gitdb: 4.0.5
     gitpython: 3.1.12
        Jinja2: 2.11.3
       libgit2: 1.4.3
      M2Crypto: Not Installed
          Mako: 1.1.4
       msgpack: 1.0.2
  msgpack-pure: Not Installed
  mysql-python: Not Installed
     pycparser: 2.19
      pycrypto: 3.14.1
  pycryptodome: 3.9.8
        pygit2: 1.9.1
        Python: 3.10.4 (main, Apr 26 2022, 19:43:24) [Clang 13.0.0 (clang-1300.0.29.30)]
  python-gnupg: 0.4.4
        PyYAML: 5.4.1
         PyZMQ: Not Installed
         smmap: 3.0.2
       timelib: 0.2.4
       Tornado: 4.5.3
           ZMQ: Not Installed

System Versions:
          dist: darwin 20.6.0
        locale: utf-8
       machine: x86_64
       release: 20.6.0
        system: Darwin
       version: 11.6.5 x86_64

Additional context
I noticed the trace message above about ip missing - brew install iproute2mac fixes this issue. But that was a wild guess, and I had to debug to even get the idea.

[Samuel Bishop]

I can confirm this behaviour on current release and latest macOS version.

edit: I can also confirm the workaround works.

[Rick Bowen]

For a zero dependency workaround, I set the enable_fqdn_grains: False configuration setting on all my macOS minions:

https://docs.saltproject.io/en/latest/ref/configuration/minion.html#enable-fqdns-grains

Maybe the default for macOS should be False just like it is for Windows?

ENABLE_FQDNS_GRAINS

Default: True

In order to calculate the fqdns grain, all the IP addresses from the minion are processed with underlying calls to socket.gethostbyaddr which can take 5 seconds to be released (after reaching socket.timeout) when there is no fqdn for that IP. These calls to socket.gethostbyaddr are processed asynchronously, however, it still adds 5 seconds every time grains are generated if an IP does not resolve. In Windows grains are regenerated each time a new process is spawned. Therefore, the default for Windows is False. In many cases this value does not make sense to include for proxy minions as it will be FQDN for the host running the proxy minion process, so the default for proxy minions is False. All other OSes default to True. This option was added in #55581.

enable_fqdns_grains: False

[jeanluc]

The workaround with iproute2mac works because its output cannot be parsed by _interfaces_ip, hence returns an empty list.

On a side note, I'm not sure why the addresses are resolved in sequence rather than in parallel on my system (still 3004.2):

[DEBUG   ] Elapsed time getting FQDNs: 32.27008390426636 seconds

This can be a seriously annoying issue when running the test suite as well.

[Andrii Nikitin]

I see it in a linux machine as well, are you sure it should have macos-specific flag?

[Gareth J. Greenaway]

The enable_fqdns_grains setting is the now the default setting for everything so this should now be resolved. It the issue still persists in the current release please open a new issue. Thanks!