HAProxy Ingress continuously reloading: tcp-services (configmap)
bootc opened this issue · comments
Description of the problem
Pods come and go in my cluster quite a lot, as one namespace in particular is used to run CI jobs for some of our projects. HAProxy Ingress notices these Pods and their associated Secrets coming and going, but notices non-existent changes to the tcp-services ConfigMap, triggering an unnecessary reload of HAProxy.
Expected behavior
The Ingress controller should see that no changes are actually needed and not reload HAProxy.
Steps to reproduce the problem
- Configure a TCP services ConfigMap using
controller.tcpin the Helm chart, for example. - Delete or create some Pods in the cluster.
dynupdate.go:104] need to reload due to config changes: [tcp-services (configmap)]
Environment information
HAProxy Ingress version: v0.14.2
Command-line options:
--configmap=tcl-ingress/haproxy-ingress
--ingress-class=haproxy
--master-socket=/var/run/haproxy/master.sock
--sort-backends
--tcp-services-configmap=tcl-ingress/haproxy-ingress-tcp
--ignore-ingress-without-class=true
--publish-service=tcl-ingress/haproxy-ingress-nlb
--wait-before-shutdown=30
Global options:
healthz-port: "10253"
https-log-format: default
prometheus-port: "9101"
stats-port: "1936"
syslog-endpoint: stdout
syslog-format: raw
tcp-log-format: default
use-haproxy-user: "true"
use-proxy-protocol: "true"
Ingress objects:
Probably not relevant.
TCP Services ConfigMap keys:
"22": gitlab/gitlab-gitlab-shell:22:PROXY
"8140": puppet/puppetserver:443:PROXY
Log from one of the updates:
I0404 12:07:37.002014 7 controller.go:338] starting haproxy update id=366
I0404 12:07:37.002096 7 converters.go:67] applying 2 change notifications: [update/Secret:gitlab-runner/runner-2gxzz8rg-project-63-concurrent-0q2tqt add/Secret:gitlab-runner/runner-2gxzz8rg-project-63-concurrent-0q2tqt]
I0404 12:07:37.002307 7 ingress.go:232] syncing 0 host(s) and 0 backend(s)
I0404 12:07:37.004877 7 instance.go:441] updating 0 host(s): []
I0404 12:07:37.004888 7 instance.go:458] updating 0 backend(s): []
I0404 12:07:37.004922 7 dynupdate.go:104] need to reload due to config changes: [tcp-services (configmap)]
I0404 12:07:37.041428 7 instance.go:401] haproxy successfully reloaded (external)
I0404 12:07:37.041479 7 controller.go:347] finish haproxy update id=366: parse_ingress=0.292283ms parse_tcp_svc=2.541356ms write_maps=0.006889ms write_config=2.515851ms reload_haproxy=34.022806ms total=39.379185ms
Couldn't reproduce the issue, but it seems doable in the case k8s endpoints are provided in a distinct order between reconciliations. I hope #1003 can address that. It'd be really helpful any info just in case you can see differences in the tcp listener configuration between restarts, e.g. a duplicated endpoint entry, endpoints our of order, etc.
v0.14.3 was just released with that improvement, please let us know if you continue to reproduce this behavior.
Sorry I never got back to you about this. Unfortunately I'm still seeing the same behaviour with quay.io/jcmoraisjr/haproxy-ingress:v0.14.3; and I think you're right it's because of the ordering - but it's a bit different after your attempted fix...
This is a diff before/after an update like the above:
--- haproxy.cfg.orig 2023-06-29 10:42:58
+++ haproxy.cfg.new1 2023-06-29 10:44:48
@@ -75,8 +75,8 @@
bind :8140 accept-proxy
mode tcp
option tcplog
- server srv003 10.198.28.230:443 check port 443 inter 2s
- server srv002 10.198.3.250:443 check port 443 inter 2s
+ server srv002 10.198.28.230:443 check port 443 inter 2s
+ server srv003 10.198.3.250:443 check port 443 inter 2s
server srv001 10.198.37.86:443 check port 443 inter 2s
# # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # #Notice how the IP addresses are indeed in sorted order and consistent, but the srv002 and srv003 entries swap between them!
Thanks for the update, really appreciated. I in fact patched the wrong place, #1011 should solve it. New tags later this week.
Thanks, I can confirm v0.14.4 and #1011 solve the issue.