docker-machine create using digitalocean driver needs droplet reboot to start listening on 2376
bitlush opened this issue · comments
I noticed that last week my deployment scripts stopped working. The main problems is:
docker-machine create "--driver" "digitalocean" [other driver settings] "droplet-name"
This errors with:
docker-machine : Error creating machine: Error running provisioning: Unable to verify the Docker daemon is
listening: Maximum number of retries (10) exceeded
This happens consistently. It seems to be because the docker host fails to listen on port 2376. I presume because docker does not start.
Rebooting the droplet after issuing docker-machine create
forces docker to start listening on port 2376 and everything then works as normal, so I have added this workaround to my deployment scripts.
How do you restart your droplet whilst it is just in creation mode by the "docker-machine create" command.
Is it possible to share the snippet of this workaround? Thanks in advance
How do you restart your droplet whilst it is just in creation mode by the "docker-machine create" command.
Is it possible to share the snippet of this workaround? Thanks in advance
The full PowerShell script workaround:
docker-machine create $docker.driver $vms.web.name
#workaround to get docker listening (was working fine for the last 3 years and only needed recently):
$droplets = doctl compute droplet list $digitalOcean.auth | ConvertFrom-Json
$droplet = $droplets | Where-Object {$_.Name -eq $vms.web.name}
doctl compute droplet-action reboot $droplet.id $digitalOcean.auth
$ip = docker-machine ip $vms.web.name
do
{
sleep 1
}
until(Test-NetConnection $ip -Port 2376 | ? { $_.TcpTestSucceeded })
So, the docker-machine create
almost works, it's just that docker fails to start listening on 2376 (presumably the last step inside docker-machine). So it's not necessary to inject a command into the docker-machine create
but just reboot the droplet afterwards and ignore the final error from docker-machine. This magically wakes up docker to start listening on port 2376. Everything below the first line of the script was unnecessary two weeks ago. It had been working fine for 3 years before this workaround was recently needed.
Note: waiting on port 2376 without rebooting does not work.
Thanks. With your insights I decided to it manually.
1 - reboot the droplet.
2 - run docker-machine regenerate-certs machine-name
About 2 months I need to execute the script again, by that time, I hope it is resolved :)
About 2 months I need to execute the script again, by that time, I hope it is resolved :)
Make sure to +1 the issue then!
For me I needed to restart the Docker service (in an Ubuntu host).
Before the service wasn't listening in the network interface:
systemctl status docker.service
● docker.service - Docker Application Container Engine
Loaded: loaded (/lib/systemd/system/docker.service; enabled; vendor preset: enabled)
Drop-In: /etc/systemd/system/docker.service.d
└─10-machine.conf
Active: active (running) since Mon 2021-01-11 16:38:43 UTC; 2min 24s ago
Docs: https://docs.docker.com
Main PID: 20707 (dockerd)
CGroup: /system.slice/docker.service
└─20707 /usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock
Jan 11 16:38:42 true dockerd[20707]: time="2021-01-11T16:38:42.849273393Z" level=warning msg="Your kernel does not support swap memory limit"
Jan 11 16:38:42 true dockerd[20707]: time="2021-01-11T16:38:42.849585773Z" level=warning msg="Your kernel does not support CPU realtime scheduler"
Jan 11 16:38:42 true dockerd[20707]: time="2021-01-11T16:38:42.850137298Z" level=info msg="Loading containers: start."
Jan 11 16:38:42 true dockerd[20707]: time="2021-01-11T16:38:42.976737533Z" level=info msg="Default bridge (docker0) is assigned with an IP address 172.17.0.0/16. Daemon option --bip can be used to set a prefe
Jan 11 16:38:43 true dockerd[20707]: time="2021-01-11T16:38:43.021144594Z" level=info msg="Loading containers: done."
Jan 11 16:38:43 true dockerd[20707]: time="2021-01-11T16:38:43.065147486Z" level=info msg="Docker daemon" commit=8891c58 graphdriver(s)=overlay2 version=20.10.2
Jan 11 16:38:43 true dockerd[20707]: time="2021-01-11T16:38:43.065334208Z" level=info msg="Daemon has completed initialization"
Jan 11 16:38:43 true systemd[1]: Started Docker Application Container Engine.
Jan 11 16:38:43 true dockerd[20707]: time="2021-01-11T16:38:43.097588922Z" level=info msg="API listen on /var/run/docker.sock"
Jan 11 16:38:46 true systemd[1]: Started Docker Application Container Engine.
After restarting docker.service
it listens to 0.0.0.0:2376
and the daemon can be contacted by docker-machine
:
systemctl restart docker.service
systemctl status docker.service
● docker.service - Docker Application Container Engine
Loaded: loaded (/lib/systemd/system/docker.service; enabled; vendor preset: enabled)
Drop-In: /etc/systemd/system/docker.service.d
└─10-machine.conf
Active: active (running) since Mon 2021-01-11 16:41:18 UTC; 2s ago
Docs: https://docs.docker.com
Main PID: 21666 (dockerd)
Tasks: 12
Memory: 41.1M
CPU: 438ms
CGroup: /system.slice/docker.service
└─21666 /usr/bin/dockerd -H tcp://0.0.0.0:2376 -H unix:///var/run/docker.sock --storage-driver overlay2 --tlsverify --tlscacert /etc/docker/ca.pem --tlscert /etc/docker/server.pem --tlskey /etc/doc
Jan 11 16:41:18 true dockerd[21666]: time="2021-01-11T16:41:18.302975014Z" level=warning msg="Your kernel does not support swap memory limit"
Jan 11 16:41:18 true dockerd[21666]: time="2021-01-11T16:41:18.303558253Z" level=warning msg="Your kernel does not support CPU realtime scheduler"
Jan 11 16:41:18 true dockerd[21666]: time="2021-01-11T16:41:18.304190321Z" level=info msg="Loading containers: start."
Jan 11 16:41:18 true dockerd[21666]: time="2021-01-11T16:41:18.483956848Z" level=info msg="Default bridge (docker0) is assigned with an IP address 172.17.0.0/16. Daemon option --bip can be used to set a prefe
Jan 11 16:41:18 true dockerd[21666]: time="2021-01-11T16:41:18.528351736Z" level=info msg="Loading containers: done."
Jan 11 16:41:18 true dockerd[21666]: time="2021-01-11T16:41:18.564827725Z" level=info msg="Docker daemon" commit=8891c58 graphdriver(s)=overlay2 version=20.10.2
Jan 11 16:41:18 true dockerd[21666]: time="2021-01-11T16:41:18.564977429Z" level=info msg="Daemon has completed initialization"
Jan 11 16:41:18 true systemd[1]: Started Docker Application Container Engine.
Jan 11 16:41:18 true dockerd[21666]: time="2021-01-11T16:41:18.607117310Z" level=info msg="API listen on [::]:2376"
Jan 11 16:41:18 true dockerd[21666]: time="2021-01-11T16:41:18.617467835Z" level=info msg="API listen on /var/run/docker.sock"
This seems to be the same problem than #4858.