Giters

medic / cht-watchdog

Configuration for deploying a monitoring/alerting stack for CHT

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

Should we be pinning default versions for the Docker images used by Watchdog?

jkuester opened this issue · comments

commented

Currently we the default version for all the Docker images used by Watchdog is just latest. This has some pros and cons:

Pros:

  • Convenient from a development standpoint since we do not have to worry about continually bumping the versions every time one of the images is upgraded
  • Could be considered convenient for a prod deployment for the same reason (although would probably not be recommended due to stability concerns).

Cons:

  • We do not have any historic record of which versions of the images actually worked with a particular version of Watchdog. (Probably could compile information by checking release dates for versions of all the images, but that would be time-consuming and error-prone)
  • Any deployments that do not set a custom version for the images will just be running against latest. If one of the images pushes a breaking change that is not supported by Watchdog it could break the deployment.
  • If a deployment does set a custom version for the images, it is not clear, when upgrading Watchdog, if/when they need to also update the associated image versions.

Proposal

I am thinking we should just set the latest versions as the default values instead of using latest. Perhaps we could add some CI checks that would remind us to bump the version (or could Dependabot be configured to just submit a PR??). This way, when you deployed Watchdog, you would not have to worry about manually managing all the image versions. Instead, if you do not have any custom versions configured, then you will automatically get the new image versions when upgrading Watchdog.

@mrjones-plip @m5r Thoughts?

commented

yes, this is a nice split between pinned versions but that still can easily be upgraded when the default changes. We'll loose the benefits of always being on latest, but gain the ability to know which docker image versions where pinned to which Watchdog version.

Ship it!

commented

+1 for more consistency/stability 🚀

commented

So it turns out that dependabot does not support monitoring docker compose files and they've been tracking this issue for 5 years now with no meaningful progress 🤷‍♂️
The alternative to make it work with dependabot is to create a Dockerfile for each of the 7 images we're depending on within the repo and point dependabot to those Dockerfiles. It adds indirection and I find it's not worth the confusion.

Another alternative to get notified when a new docker image of a container we're running is available that we could run alongside watchdog like https://github.com/containrrr/watchtower. I'm not comfortable with this solution because it requires giving access to the docker socket /var/run/docker.sock and that would give it way too much control over our deployment.

Last resort would be to use an external service like https://docker-notify.com/ or https://releasealert.dev/

For now I would say let's keep checking manually every now and then when new image tags are available, for example with regctl https://github.com/regclient/regclient. If we notice it's getting too cumbersome to manage, we can reconsider the options listed above (+ anything that comes up between now and then!)
In the meantime, I'll open a subsequent PR with available upgrades so far.

commented

👍 Sounds good! Really, I imagine it would not be too difficult to roll-our-own GitHub action that would just compare our current version of the Docker images with the latest from DockerHub. 🤔 Not a big deal to do it manually for now, though!