Long delay when printing catalog server in Puppet 7.28.0 on IPv6-only clients
sagepe opened this issue · comments
Describe the Bug
When running Puppet agent 7.28 interactively (e.g. puppet agent --test --noop) on a host with an IPv6 address but without an IPv4 address, the run consistently hangs for over two minutes after printing Info: Loading facts and before printing Notice: Requesting catalog from puppet.example.com:8140.
When running on a dual stack host with both IPv4 and IPv6 addresses, there is no pause and the Notice includes an IPv4 address in parentheses: Notice: Requesting catalog from puppet.example.com:8140 (10.0.0.1).
Expected Behavior
There should be no delay when running Puppet on hosts with only an IPv6 address.
Steps to Reproduce
On an IPv6 only host that uses a Puppet server with both IPv4 and IPv6 addresses, run puppet agent --test --noop.
Environment
- Version 7.28.0
- Platform Debian Bullseye
Additional Context
The Puppet server is dual stack with routable IPv4 and IPv6 addresses. These are in DNS and can be resolved from the client system using the system resolver as normal.
This delay was not present in previous versions of Puppet. Reverting Puppet agent to 7.27.0 resolves the problem.
Looking at puppetlabs/puppet#9126, it appears that the lookup using Resolv.getaddressis causing the delay.
Please note that all hostnames and IP addresses have been redacted and replaced in the examples below, but otherwise they are reproduced exactly as we ran them.
Using Ruby 2.7.8 on a dual stack host:
$ ruby --version; time ruby -r resolv -e 'puts Resolv.getaddress("puppet.example.com")'
ruby 2.7.8p225 (2023-03-30 revision 1f4d455848) [x86_64-linux]
10.0.0.1
real 0m0.162s
user 0m0.137s
sys 0m0.024s
Using Ruby 2.7.8 on an IPv6 only host:
$ ruby --version; time ruby -r resolv -e 'puts Resolv.getaddress("puppet.example.com")'
ruby 2.7.8p225 (2023-03-30 revision 1f4d455848) [x86_64-linux]
Traceback (most recent call last):
2: from -e:1:in `<main>'
1: from /opt/rbenv/versions/2.7.8/lib/ruby/2.7.0/resolv.rb:44:in `getaddress'
/opt/rbenv/versions/2.7.8/lib/ruby/2.7.0/resolv.rb:94:in `getaddress': no address for puppet.example.com (Resolv::ResolvError)
real 2m40.428s
user 0m0.182s
sys 0m0.056s
Using Ruby 3.2.2 on an IPv6 only host does return a result, but it's an IPv4 address which is arguably not what you would expect, so I suspect this issue is less visible in Puppet 8.x but may represent a related but slightly different bug:
$ ruby --version; time ruby -r resolv -e 'puts Resolv.getaddress("puppet.example.com")'
ruby 3.2.2 (2023-03-30 revision e51014f9c0) [x86_64-linux]
10.0.0.1
real 0m0.191s
user 0m0.152s
sys 0m0.039s
Noting that if you use the Resolv gem under Ruby 2.7.8 rather than the builtin, this works as per 3.2.2 which would seem to be what you'd expect:
$ cat Gemfile; ruby --version; time ruby -r 'bundler/setup' -r resolv -e 'puts Resolv.getaddress("puppet.example.com")'
source "https://rubygems.org"
gem "resolv", "0.3.0"
ruby 2.7.8p225 (2023-03-30 revision 1f4d455848) [x86_64-linux]
10.0.0.1
real 0m0.328s
user 0m0.296s
sys 0m0.031s
Thanks for letting us know @sagepe. Are you able to provide the steps you used to disable IPv4 on a dual stack machine so we can try to reproduce this? Also the output of facter networking would be helpful.
Are you able to provide the steps you used to disable IPv4 on a dual stack machine so we can try to reproduce this?
I just provision one in our infra, so I don't manually make the changes. It's a bit tricky to provide definitive instructions as it will of course vary depending on how the server instances are provisioned, the OS family, etc.
Our IPv6 only systems do have an IPv4 loopback, but no IPv4 address on their public interfaces. They have an IPv6 loopback address, a single global public IPv6 address and a link-local IPv6 address - see the facter output; I've redacted the actual addresses and hostnames:
If you're starting point is a dual stack system with both IPv4 and IPv6 addresses, you may be able to replicate it with a /etc/network/interfaces file that looks something like the below; you'll also need to ensure that you have functioning IPv6 DNS resolvers configured in /etc/resolv.conf and no automation that will attempt to override/revert things (including a DHCP service or systemd of any of the other things that might think it knows better).
auto lo
iface lo inet loopback
auto eth0
iface eth0 inet6 static
address <YOUR IPv6 address>
Migrated issue to PUP-12022
We've added an option to disable this behavior, avoiding the DNS lookup altogether. I'll leave this open since it seems we should be able to do the lookup with a shorter timeout.
Since there is an option to disable this behavior and this impacts 7.x which is only getting security updates because it is LTS, this issue will be closed.