[Scaleway] Nodes cannot pull image from docker registry
kotluk opened this issue · comments
Hey, I have an issue that don't know how to investigate further and how to solve it. I tried it a few times by removing all VPS and started again but every time I end up with the same error.
I am trying to set up my cluster on Scaleway. I follow instructions one by one, and I am at the point where I installed wave as CNI, and I've got:
kube-system weave-net-dtwbj 2/2 Running 1 9d
kube-system weave-net-kmxq7 0/2 Init:ImagePullBackOff 0 9d
kube-system weave-net-pzfcj 0/2 Init:ImagePullBackOff 0 9d
So my issue is on my nodes but not on master.
I found this #74 and this time I applied these suggestions, but the output is the same.
UFW / Firewall
I skip the part with firewall, on every VPS I've got:
> ufw status
Status: inactive
In scaleway config all my VPS have the same security policy applied. Only outbound traffic on ports [25, 465, 587] is dropping.
Internet connection
On both my nodes I've issue to download images from docker's registry and I believe that this is the real issue here
> docker pull hello-world
Using default tag: latest
Error response from daemon: Get https://registry-1.docker.io/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
On master hello-world was pulled successfully.
Nodes have internet connection outside:
--- google.com ping statistics ---
9 packets transmitted, 9 received, 0% packet loss, time 8011ms
rtt min/avg/max/mdev = 1.008/1.139/1.258/0.073 ms
WireGuard
By the output of wg show I assume that VPN between my VPS is set up correctly
peer: 3
endpoint: 3-priv IP:51820
allowed ips: 10.0.1.3/32
latest handshake: 1 minute, 17 seconds ago
transfer: 7.50 GiB received, 6.50 GiB sent
peer: 2
endpoint: 2-priv IP:51820
allowed ips: 10.0.1.2/32
latest handshake: 1 minute, 41 seconds ago
transfer: 4.96 GiB received, 6.11 GiB sent
Could anybody help me track the issue down and help me to fix it? I can provide any kinds of logs you wish just tell me how I can get it
I‘ll try to have a look at the suggestions made in #74, but I’m really busy at the moment.
Thank you, no pressure. Once more I decided to remove all my VPS and try again but with this error in mind. I installed docker firstly and then follow this guide. After each bigger step I run docker pull image where image was every time different (PHP, nginx etc.) to be sure that it has to download it from registry and will not use any cache. After this process I find out that wireguard cause the issue.
My VPS's config looked like this:
1 VPS with public IP
2 VPS without public IP
And the one with public IP worked well, and two others didn't, so I added at Scalaway public IP's, and it starts working (docker pull). Unfortunately I as well didn't have enough time to try the rest of the guide. Hopefully in this weekend I will try and let you know.
I don't understand why VPS worked well without WG and public IP and after setting up WG I suddenly need public IP. If you have any idea let me know.
Right now treat this as on hold I will leave here information if adding public IP's solves my problems :)
That’s interesting! The provisioning repository does enable public ips on all hosts, that’s why I‘ve never experienced this issue. Keep me posted.
Hey I managed to set up cluster :) Not adding IP addresses to all my nodes was a mistake. But I still don't understand why setting up WireGuard messed networking on VPS. I followed additional instructions from #74 and didn't have to do any ip route from guide and anything connected to that so if you will find any spare time maybe it's worth your attention, and if it doesn't break anything else, IMO it will be good to mention it in guide as second option :)
Thank you very much for this guide it is very useful