Unhealthy maildev container in docker compose
Nikola-Milovic opened this issue · comments
➜ docker ps | grep mail
7a82025c0d2e maildev/maildev "bin/maildev" 4 minutes ago Up 4 minutes (unhealthy) 1025/tcp, 1080/tcp maildev-1
➜ docker compose logs -f maildev
maildev-1 | MailDev using directory /tmp/maildev-1
maildev-1 | MailDev webapp running at http://0.0.0.0:1080/
maildev-1 | MailDev SMTP Server running at 0.0.0.0:1025
My docker compose file
maildev:
image: maildev/maildev:2.1.0
networks:
- platform
Docker Compose version v2.25.0
My maildev stopped being tracked by traefik since I updated docker compose, so it might be related not sure. A bit of debugging and I found that the container is unhealthy. Considering how barebones my setup is, I am guessing its something related to maildev itself
{
"Status": "unhealthy",
"FailingStreak": 5,
"Log": [
{
"Start": "2024-03-21T09:52:10.217941825+01:00",
"End": "2024-03-21T09:52:10.295743634+01:00",
"ExitCode": 1,
"Output": "Connecting to localhost:1080 ([::1]:1080)\nwget: can't connect to remote host: Connection refused\n"
},
{
"Start": "2024-03-21T09:52:20.297273607+01:00",
"End": "2024-03-21T09:52:20.37955279+01:00",
"ExitCode": 1,
"Output": "Connecting to localhost:1080 ([::1]:1080)\nwget: can't connect to remote host: Connection refused\n"
},
{
"Start": "2024-03-21T09:52:30.380291256+01:00",
"End": "2024-03-21T09:52:30.458289518+01:00",
"ExitCode": 1,
"Output": "Connecting to localhost:1080 ([::1]:1080)\nwget: can't connect to remote host: Connection refused\n"
},
{
"Start": "2024-03-21T09:52:40.458959601+01:00",
"End": "2024-03-21T09:52:40.563597174+01:00",
"ExitCode": 1,
"Output": "Connecting to localhost:1080 ([::1]:1080)\nwget: can't connect to remote host: Connection refused\n"
},
{
"Start": "2024-03-21T09:52:50.564735793+01:00",
"End": "2024-03-21T09:52:50.628254729+01:00",
"ExitCode": 1,
"Output": "Connecting to localhost:1080 ([::1]:1080)\nwget: can't connect to remote host: Connection refused\n"
}
]
}
Can someone try and confirm that its working or not working for them as well?
Same issue here. It works for me locally (MacOS) and appears healthy. At CI though I receive the same "MailDev SMTP Server running at 0.0.0.0:1025" success message, but the container still shows as unhealthy. Can it be something related to changes in docker daemon or something like that? Also, maildev/maildev image hasn't changed in 8 months apparently
@pallost Seems that its docker related, my services are also having weird dns issues, and they cannot resolve any address
For us:
- maildev container works locally in docker compose (both arm64 and amd64 images)
- maildev container does not work in gitlab CI in docker compose (it logs that the UI would have started but its own health check fails)
- maildev container works in gitlab CI when started as a sidecar (without environment variables)
I ended up removing maildev from the docker compose setup because realized that we do not need it (yet) for our e2e tests. The sidecar version without any environment variables still works and can be used with the unit tests.
I absolutely don't think this is related to maildev, really feels like there's some networking issue within the docker service itself. Good luck for further debugging!
Downgrade docker-ce to older release (25.0.5) fix the issue.
Same issue. As a workaround I put following in the docker compose file:
healthcheck:
test: 'wget -O - http://127.0.0.1:$${MAILDEV_WEB_PORT}$${MAILDEV_BASE_PATHNAME}/healthz || exit 1'
Got same issue, seems more a recent bugfix in docker that starts to use the planned default healthcheck configured in the image.
Healthcheck configured doesn't work and docker does its job:
docker compose exec -it maildev wget -q http://127.0.0.1/healthz
wget: can't open 'healthz': File exists
I couldn't find any wget call that returns 200 code after beeing called a second time (always says this strange error can't open ...).
My fix was to force disabling healthcheck like this:
healthcheck:
disable: true
If you use docker compose up -d --wait command, you have to fake a healthy check. This could be performed like this:
healthcheck:
test: ["CMD", "touch", "health"]
interval: 15s
timeout: 5s
retries: 5
Root cause is tracked here: #474
@zguig52 I think my specific issue is different from the one you mentioned, mine seems to be related to docker itself and some update in v26.0 (since for me more than maildev and healthcheck are affected). But I am leaving the issue open since others seem to have similar issues
@Nikola-Milovic , got this also when upgrading Rocky 9 to latest docker:
docker --version
Docker version 26.0.0, build 2ae903e
I got other issues with curl/wget localhost configured healthchecks. I had to replace localhost with 127.0.0.1. Other communications are based on self defined extra_hosts.
Seems there is also something with DNS resolution in this version.
Could be related to this issue: moby/moby#47414
I had also some weird transient behavior recently even before this upgrade with v25 where sometimes my containers where unable to communicate between them after restart and I had to destruct them to restart them (multiple networks attached with self configured IPs through IPAM module). I saw that they did not had all networks instantiated thus not listening on expected IPs.
I can confirm that it does not work with Docker version 26.0.0, build 2ae903e. It works with Docker version 25.0.5, build 5dc9bcc.
The root cause is that the wget (for healthcheck) tries ipv6 to connect.
From within the maildev container:
~ $ wget http://localhost:1080/
Connecting to localhost:1080 ([::1]:1080)
wget: can't connect to remote host: Connection refused
~ $ wget http://127.0.0.1:1080/
Connecting to 127.0.0.1:1080 (127.0.0.1:1080)
saving to 'index.html'
index.html 100% |*************************************************************************************************************************************************************************************************************************************************************| 8390 0:00:00 ETA
'index.html' saved
Once can confirm that using telnet (after installing it):
telnet -6 localhost 1080
Trying ::1...
telnet: Unable to connect to remote host: Connection refused
Same problem, and also related to localhost being translated as ipv6 instead of ipv4...
I've also defined my health check to support a self-signed certificate to run it over https:
healthcheck:
test: wget --no-check-certificate --user-agent='Docker healthcheck' -O /dev/null https://localhost:1080/healthz || exit 1
interval: 10s
timeout: 1sNot a big deal, but it might be worth adding it to the documentation?
This issue is also present on Docker 20.10.24+dfsg1 (Debian 12) when IPv6 is manually enabled. It's possible to reproduce it by placing the following lines in your /etc/docker/daemon.json,
{
"ipv6": true,
"fixed-cidr-v6": "fd00::/64"
}Maildev currently listens on IPv4 only:
~ $ netstat -lptn
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
tcp 0 0 0.0.0.0:1025 0.0.0.0:* LISTEN 1/node
tcp 0 0 0.0.0.0:1080 0.0.0.0:* LISTEN 1/node
Maybe it should listen on IPv6 too
is this fixed on laster image?