ingress watch causes repeated restarts on pfsense services?
bezerker opened this issue · comments
Hi,
I'm using ingress watching with unbound. It works well, except, both ingresses I use (nginx and traefik) seem to cause restarts over and over.
Traefik will update the status endpoint every minute. Even though nothing changed, this triggers a restart. (Kind of expected). With nginx, this update must occur less often, as it seems roughly every 10 minutes or so.
Is there a way to make the ingress watch do a diff before triggering a restart? If not, I can look into trying to code something in, but not sure if this is a known issue.
I'm using 0.1.7 and also tried 0.1.5.
Thanks again for a great controller! Sorry to bug with an issue. :(
Example logs from traefik triggered ingresses.
019-11-29T18:56:44+00:00 plugin (pfsense-dns-ingresses): successfully reloaded DHCP service
2019-11-29T18:57:24+00:00 plugin (pfsense-dns-ingresses): setting hostname entry: Host - kibana.k8s.bezerker.org, IP - 192.168.30.20
2019-11-29T18:57:24+00:00 plugin (pfsense-dns-ingresses): setting hostname entry: Host - traefik-dashboard.k8s.bezerker.org, IP - 192.168.30.20
2019-11-29T18:57:34+00:00 plugin (pfsense-dns-ingresses): successfully reloaded unbound service
2019-11-29T18:57:35+00:00 plugin (pfsense-dns-ingresses): successfully reloaded DHCP service
2019-11-29T18:58:24+00:00 plugin (pfsense-dns-ingresses): setting hostname entry: Host - kibana.k8s.bezerker.org, IP - 192.168.30.20
2019-11-29T18:58:24+00:00 plugin (pfsense-dns-ingresses): setting hostname entry: Host - traefik-dashboard.k8s.bezerker.org, IP - 192.168.30.20
2019-11-29T18:58:33+00:00 plugin (pfsense-dns-ingresses): successfully reloaded unbound service
2019-11-29T18:58:35+00:00 plugin (pfsense-dns-ingresses): successfully reloaded DHCP service
2019-11-29T18:59:24+00:00 plugin (pfsense-dns-ingresses): setting hostname entry: Host - kibana.k8s.bezerker.org, IP - 192.168.30.20
2019-11-29T18:59:24+00:00 plugin (pfsense-dns-ingresses): setting hostname entry: Host - traefik-dashboard.k8s.bezerker.org, IP - 192.168.30.20
2019-11-29T18:59:33+00:00 plugin (pfsense-dns-ingresses): successfully reloaded unbound service
2019-11-29T18:59:35+00:00 plugin (pfsense-dns-ingresses): successfully reloaded DHCP service
2019-11-29T19:00:24+00:00 plugin (pfsense-dns-ingresses): setting hostname entry: Host - kibana.k8s.bezerker.org, IP - 192.168.30.20
2019-11-29T19:00:24+00:00 plugin (pfsense-dns-ingresses): setting hostname entry: Host - traefik-dashboard.k8s.bezerker.org, IP - 192.168.30.20
2019-11-29T19:00:33+00:00 plugin (pfsense-dns-ingresses): successfully reloaded unbound service
2019-11-29T19:00:35+00:00 plugin (pfsense-dns-ingresses): successfully reloaded DHCP service
2019-11-29T19:01:24+00:00 plugin (pfsense-dns-ingresses): setting hostname entry: Host - kibana.k8s.bezerker.org, IP - 192.168.30.20
2019-11-29T19:01:24+00:00 plugin (pfsense-dns-ingresses): setting hostname entry: Host - traefik-dashboard.k8s.bezerker.org, IP - 192.168.30.20
2019-11-29T19:01:34+00:00 plugin (pfsense-dns-ingresses): successfully reloaded unbound service
2019-11-29T19:01:35+00:00 plugin (pfsense-dns-ingresses): successfully reloaded DHCP service
2019-11-29T19:02:24+00:00 plugin (pfsense-dns-ingresses): setting hostname entry: Host - kibana.k8s.bezerker.org, IP - 192.168.30.20
2019-11-29T19:02:24+00:00 plugin (pfsense-dns-ingresses): setting hostname entry: Host - traefik-dashboard.k8s.bezerker.org, IP - 192.168.30.20
2019-11-29T19:02:34+00:00 plugin (pfsense-dns-ingresses): successfully reloaded unbound service
2019-11-29T19:02:35+00:00 plugin (pfsense-dns-ingresses): successfully reloaded DHCP service
2019-11-29T19:03:24+00:00 plugin (pfsense-dns-ingresses): setting hostname entry: Host - kibana.k8s.bezerker.org, IP - 192.168.30.20
2019-11-29T19:03:24+00:00 plugin (pfsense-dns-ingresses): setting hostname entry: Host - traefik-dashboard.k8s.bezerker.org, IP - 192.168.30.20
2019-11-29T19:03:34+00:00 plugin (pfsense-dns-ingresses): successfully reloaded unbound service
2019-11-29T19:03:35+00:00 plugin (pfsense-dns-ingresses): successfully reloaded DHCP service
2019-11-29T19:04:24+00:00 plugin (pfsense-dns-ingresses): setting hostname entry: Host - kibana.k8s.bezerker.org, IP - 192.168.30.20
2019-11-29T19:04:24+00:00 plugin (pfsense-dns-ingresses): setting hostname entry: Host - traefik-dashboard.k8s.bezerker.org, IP - 192.168.30.20
2019-11-29T19:04:33+00:00 plugin (pfsense-dns-ingresses): successfully reloaded unbound service
2019-11-29T19:04:34+00:00 plugin (pfsense-dns-ingresses): successfully reloaded DHCP service
2019-11-29T19:05:24+00:00 plugin (pfsense-dns-ingresses): setting hostname entry: Host - kibana.k8s.bezerker.org, IP - 192.168.30.20
2019-11-29T19:05:24+00:00 plugin (pfsense-dns-ingresses): setting hostname entry: Host - traefik-dashboard.k8s.bezerker.org, IP - 192.168.30.20
2019-11-29T19:05:34+00:00 plugin (pfsense-dns-ingresses): successfully reloaded unbound service
2019-11-29T19:05:35+00:00 plugin (pfsense-dns-ingresses): successfully reloaded DHCP service
Ah interesting. I didn't use that one long enough to see that behavior but I can dig into the code and see what's up. Certainly making sure the IP actually updated is a better approach.
Try v0.1.9 and let me know.
Hrm, looks like traefik is still kicking it.
2019-11-30T16:23:25+00:00 store successfully initialized
2019-11-30T16:23:25+00:00 waiting for ConfigMap kube-system/kubernetes-pfsense-controller-config to be present and valid
2019-11-30T16:23:30+00:00 controller config loaded/updated
2019-11-30T16:23:46+00:00 controller config loaded/updated
2019-11-30T16:23:46+00:00 loading plugin pfsense-dns-ingresses
2019-11-30T16:23:57+00:00 plugin (pfsense-dns-ingresses): setting hostname entry: Host - kibana.k8s.bezerker.org, IP - 192.168.30.20
2019-11-30T16:23:57+00:00 plugin (pfsense-dns-ingresses): setting hostname entry: Host - traefik-dashboard.k8s.bezerker.org, IP - 192.168.30.20
2019-11-30T16:24:07+00:00 plugin (pfsense-dns-ingresses): successfully reloaded unbound service
2019-11-30T16:24:08+00:00 plugin (pfsense-dns-ingresses): successfully reloaded DHCP service
2019-11-30T16:24:13+00:00 plugin (pfsense-dns-ingresses): /apis/networking.k8s.io/v1beta1/namespaces/traefik-ingress/ingresses/traefik-dashboard MODIFIED - 25410332
2019-11-30T16:24:13+00:00 plugin (pfsense-dns-ingresses): /apis/networking.k8s.io/v1beta1/namespaces/traefik-ingress/ingresses/traefik-dashboard MODIFIED - 25410333
2019-11-30T16:24:24+00:00 plugin (pfsense-dns-ingresses): setting hostname entry: Host - kibana.k8s.bezerker.org, IP - 192.168.30.20
2019-11-30T16:24:24+00:00 plugin (pfsense-dns-ingresses): setting hostname entry: Host - traefik-dashboard.k8s.bezerker.org, IP - 192.168.30.20
2019-11-30T16:24:34+00:00 plugin (pfsense-dns-ingresses): successfully reloaded unbound service
2019-11-30T16:24:35+00:00 plugin (pfsense-dns-ingresses): successfully reloaded DHCP service
2019-11-30T16:25:13+00:00 plugin (pfsense-dns-ingresses): /apis/networking.k8s.io/v1beta1/namespaces/traefik-ingress/ingresses/traefik-dashboard MODIFIED - 25410465
2019-11-30T16:25:13+00:00 plugin (pfsense-dns-ingresses): /apis/networking.k8s.io/v1beta1/namespaces/traefik-ingress/ingresses/traefik-dashboard MODIFIED - 25410466
2019-11-30T16:25:24+00:00 plugin (pfsense-dns-ingresses): setting hostname entry: Host - kibana.k8s.bezerker.org, IP - 192.168.30.20
2019-11-30T16:25:24+00:00 plugin (pfsense-dns-ingresses): setting hostname entry: Host - traefik-dashboard.k8s.bezerker.org, IP - 192.168.30.20
2019-11-30T16:25:34+00:00 plugin (pfsense-dns-ingresses): successfully reloaded unbound service
2019-11-30T16:25:35+00:00 plugin (pfsense-dns-ingresses): successfully reloaded DHCP service
2019-11-30T16:26:03+00:00 controller config loaded/updated
(last config update was me disabling ingress again)
Interestingly this appaers to be due to traefik updating the status endpoint every minute
from the traefik logs:
{"level":"info","msg":"Updated status on ingress traefik-ingress/traefik-dashboard","time":"2019-11-30T16:14:13Z"}
{"level":"info","msg":"Updated status on ingress traefik-ingress/traefik-dashboard","time":"2019-11-30T16:15:13Z"}
{"level":"info","msg":"Updated status on ingress traefik-ingress/traefik-dashboard","time":"2019-11-30T16:16:13Z"}
{"level":"info","msg":"Updated status on ingress traefik-ingress/traefik-dashboard","time":"2019-11-30T16:17:13Z"}
{"level":"info","msg":"Updated status on ingress traefik-ingress/traefik-dashboard","time":"2019-11-30T16:18:13Z"}
{"level":"info","msg":"Updated status on ingress traefik-ingress/traefik-dashboard","time":"2019-11-30T16:19:14Z"}
{"level":"info","msg":"Updated status on ingress traefik-ingress/traefik-dashboard","time":"2019-11-30T16:20:13Z"}
{"level":"info","msg":"Updated status on ingress traefik-ingress/traefik-dashboard","time":"2019-11-30T16:21:13Z"}
{"level":"info","msg":"Updated status on ingress traefik-ingress/traefik-dashboard","time":"2019-11-30T16:22:13Z"}
{"level":"info","msg":"Updated status on ingress traefik-ingress/traefik-dashboard","time":"2019-11-30T16:23:13Z"}
{"level":"info","msg":"Updated status on ingress traefik-ingress/traefik-dashboard","time":"2019-11-30T16:24:13Z"}
{"level":"info","msg":"Updated status on ingress traefik-ingress/traefik-dashboard","time":"2019-11-30T16:25:13Z"}
{"level":"info","msg":"Updated status on ingress traefik-ingress/traefik-dashboard","time":"2019-11-30T16:26:14Z"}
{"level":"info","msg":"Updated status on ingress traefik-ingress/traefik-dashboard","time":"2019-11-30T16:27:13Z"}
Honestly, i'll bet if i disable dashboard this will become far less frequent, but it was still occurring at random intervals with nginx-ingress as well. (I'm not sure if it's normal for traefik to update status every minute like that for dashboard.)
Sure you updated? I coded it to only reload the services when IP changes...I assume when they 'update' it's a noop as the IP is the same (I'm using traefik 1.x and confirmed it didn't update when the IP didn't actually change)?
Indeed. updated to 0.1.9.
kubernetes-pfsense-controller:
Container ID: docker://0aa68d9b62b50997170dc5f3cfe7b492fd9a553cd5e72a898fcafc778ed849d1
Image: docker.io/travisghansen/kubernetes-pfsense-controller:v0.1.9
Image ID: docker-pullable://travisghansen/kubernetes-pfsense-controller@sha256:50e77db1d7454f28515d46bcd4e4e3404d646de1d27f7e393b2d97ce8c825aef
Port: <none>
Host Port: <none>
State: Running
Started: Sat, 30 Nov 2019 11:23:25 -0500
Interesting on traefik. I'm also using 1.x and seeing the above in my logs. Granted, there are ojnly two ingresses configured on my cluster currently.
Are you seeing similar update spam in your logs for traefik? Perhaps it's an issue with my traefik configuration that I'm not seeing.
Yeah, image looks legit. Not really sure how/why it's trigger a reload. In my setup I'm not seeing it update that frequent but I didn't watch it that closely. I was manually triggering the update of ingresses by simply changing an annotation repeatedly and observed that updates did not trigger unless the IP changed (which I did trigger via toggling some metallb settings back and forth to ensure it would update when appropriate).
Any chance you could run the code directly? And do some minor tweaks to debug stuff a little?
Any further luck/issues with this one?