Do not include mongo in liveness probe state
antechrestos opened this issue · comments
Currently by default mongo state is reported in liveness state. When connection is lost, a call to /actuator/health/liveness hangs until mongo time out is reached (30s by default).
As this endpoint is quite often used in k8s for liveness probe, this will lead to scenario quite problematic as follows
- the mongo cluster is out of reach
- the liveness probe answers badly (timeout or bad status)
- kubelet restarts the process
As the mongo is the same for all pod, this will lead to the cluster restarting all its pods as their liveness probe is broken, consuming quite cpu, memory, and so on.
Moreover the spring documentation states that
By default, Spring Boot does not add other health indicators to these groups.
While I understand the idea and I may understand the fact that a pod should not receive web trafic while its connection to the database is broken, this would lead to a readidess probe , not a liveness probe
Moreover, why running a ping command for each http request and not return the last observed state? This is what causes the timeout
Thank you @antechrestos for getting in touch. Actuator is part of the Spring Boot project, so I kindly ask you to report the issue there.
Please make sure to also provide a complete minimal sample (something that we can unzip or git clone, build, and deploy) that reproduces the problem.
Thank you!
@christophstrobl that is not you who chose to expose it on actuator then? You just provided the health indicator and actuator wrongly exposed it in liveness probe?
thank you for the quick reply by the way