Deployed PODs have no access to internet

Question

Deployed PODs have no access to internet

luisfn opened this issue 2 years ago · comments

First of all, thank you very much for this work, it made my life much much easier, it did created the kubernetes cluster for me with no effort.

That being said, it seems in my case that my deployed pods can access external APIs/Urls (Telegram for example). It seems to be a DNS issue due to my first tries to fix.

Here is what I tried:

k get pods -n kube-system

NAME                            READY   STATUS    RESTARTS      AGE
coredns-787d4945fb-sdtvf        1/1     Running   1 (11h ago)   13h
coredns-787d4945fb-tkr82        1/1     Running   1 (11h ago)   13h
etcd-node1                      1/1     Running   1 (11h ago)   13h
kube-apiserver-node1            1/1     Running   1 (11h ago)   13h
kube-controller-manager-node1   1/1     Running   1 (11h ago)   13h
kube-proxy-466ld                1/1     Running   1 (11h ago)   13h
kube-proxy-8jn6q                1/1     Running   1 (11h ago)   13h
kube-proxy-bf5mt                1/1     Running   1 (11h ago)   13h
kube-proxy-xdlwk                1/1     Running   1 (11h ago)   13h
kube-scheduler-node1            1/1     Running   1 (11h ago)   13h
weave-net-k5b9d                 2/2     Running   2 (11h ago)   13h
weave-net-lklhq                 2/2     Running   3 (11h ago)   13h
weave-net-rzjhn                 2/2     Running   2 (11h ago)   13h
weave-net-xcdf4                 2/2     Running   3 (11h ago)   13h

As it shows, coredns is up and running, but the log is not OK, I see a lot of no route to host messages

kubectl logs -f coredns-787d4945fb-sdtvf -n kube-system

.:53
[INFO] plugin/reload: Running configuration SHA512 = 591cf328cccc12bc490481273e738df59329c62c0b729d94e8b61db9961c2fa5f046dd37f1cf888b953814040d180f52594972691cd6ff41be96639138a43908
CoreDNS-1.9.3
linux/arm64, go1.18.2, 45b0a11
[ERROR] plugin/errors: 2 6316313032496278710.1919252235580212937. HINFO: read udp 10.32.0.3:56551->169.254.169.254:53: read: no route to host
[ERROR] plugin/errors: 2 6316313032496278710.1919252235580212937. HINFO: read udp 10.32.0.3:49321->169.254.169.254:53: read: no route to host
[ERROR] plugin/errors: 2 6316313032496278710.1919252235580212937. HINFO: read udp 10.32.0.3:44002->169.254.169.254:53: read: no route to host
[ERROR] plugin/errors: 2 checkpoint-api.weave.works. A: read udp 10.32.0.3:58720->169.254.169.254:53: i/o timeout
[ERROR] plugin/errors: 2 checkpoint-api.weave.works. AAAA: read udp 10.32.0.3:36946->169.254.169.254:53: read: no route to host

So I deployed a pod called dnsutils to see what is going on, but it had the same problem as the logs shown.

kubectl exec -it dnsutils -- nslookup google.com

;; connection timed out; no servers could be reached

command terminated with exit code 1

And this is what I have on /etc/resolv.conf (Seems ok to me)

kubectl exec -it dnsutils -- cat /etc/resolv.conf

search default.svc.cluster.local svc.cluster.local cluster.local
nameserver 10.96.0.10
options ndots:5

From the pod I want to deploy (a telegram bot), I get this:

panic: Post "https://api.telegram.org/mytoken/getMe": dial tcp: i/o timeout

I don't know what else I could try. Do you have any idea what could be the issue?

Thanks in advance!

Jérôme Petazzoni · Answer 1 · Wed Mar 01 2023 17:50:36 GMT+0800 (China Standard Time)

Hi,

There are a couple of things that you can try.

check the internal IP address of the nodes with e.g. kubectl get nodes -o wide
check the internal IP address of some pods with e.g. kubectl get pods --all-namespaces -o wide
start an Alpine test Pod, e.g. kubectl exec testpod --rm -it --image alpine
from the test Pod, ping a node with its IP address (does it work?)
from the test Pod, ping another Pod with its IP address (does it work?)
from the test Pod, check connectivity to outside targets, e.g. ping 1.1.1.1 or ping 8.8.8.8

Let us know the results!

César Augusto Fior · Answer 2 · Mon Apr 17 2023 17:38:05 GMT+0800 (China Standard Time)

Bom dia, estou com o mesmo problema @luisfn , conseguiu alguma solução?
De dentro dos pods não consigo sair para a internet!

Jérôme Petazzoni · Answer 3 · Thu May 04 2023 16:41:18 GMT+0800 (China Standard Time)

Oi! Você tentou os comandos listados acima e pode indicar suas saídas? Obrigado!

César Augusto Fior · Answer 4 · Sat May 13 2023 07:22:58 GMT+0800 (China Standard Time)

Hi,

There are a couple of things that you can try.

check the internal IP address of the nodes with e.g. kubectl get nodes -o wide

check the internal IP address of some pods with e.g. kubectl get pods --all-namespaces -o wide

start an Alpine test Pod, e.g. kubectl exec testpod --rm -it --image alpine

from the test Pod, ping a node with its IP address (does it work?)

from the test Pod, ping another Pod with its IP address (does it work?)

from the test Pod, check connectivity to outside targets, e.g. ping 1.1.1.1 or ping 8.8.8.8

Let us know the results!

Boa noite, desculpa a demora, realizei os testes qual comentou porém sem sucesso para pingar algum dos nós
resultados abaixo

NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME node1 Ready,SchedulingDisabled control-plane 5h7m v1.27.1 10.0.0.11 <none> Ubuntu 22.04.2 LTS 5.15.0-1033-oracle containerd://1.6.21 node2 Ready <none> 5h7m v1.27.1 10.0.0.12 <none> Ubuntu 22.04.2 LTS 5.15.0-1033-oracle containerd://1.6.21 node3 Ready <none> 5h7m v1.27.1 10.0.0.13 <none> Ubuntu 22.04.2 LTS 5.15.0-1033-oracle containerd://1.6.21 node4 Ready,SchedulingDisabled <none> 5h7m v1.27.1 10.0.0.14 <none> Ubuntu 22.04.2 LTS 5.15.0-1033-oracle containerd://1.6.21

PING 10.0.0.12 (10.0.0.12): 56 data bytes
^C
--- 10.0.0.12 ping statistics ---
10 packets transmitted, 0 packets received, 100% packet loss

/ # ping 1.1.1.1
PING 1.1.1.1 (1.1.1.1): 56 data bytes
^C
--- 1.1.1.1 ping statistics ---
3 packets transmitted, 0 packets received, 100% packet loss

César Augusto Fior · Answer 5 · Fri May 26 2023 04:44:27 GMT+0800 (China Standard Time)

Boa tarde, realizei um terraform destroy, atualizei o provider e estou tentando criar o ambiente novamente mas esta apresentando essa falha
╷
│ Error: remote-exec provisioner error
│
│ with oci_core_instance.["1"],
│ on main.tf line 51, in resource "oci_core_instance" "":
│ 51: provisioner "remote-exec" {
│
│ error executing "/tmp/terraform_1595989376.sh": Process exited with status 1
╵
╷
│ Error: remote-exec provisioner error
│
│ with oci_core_instance.["2"],
│ on main.tf line 51, in resource "oci_core_instance" "":
│ 51: provisioner "remote-exec" {
│
│ error executing "/tmp/terraform_2029325046.sh": Process exited with status 1
╵
╷
│ Error: remote-exec provisioner error
│
│ with oci_core_instance.["3"],
│ on main.tf line 51, in resource "oci_core_instance" "":
│ 51: provisioner "remote-exec" {
│
│ error executing "/tmp/terraform_154756949.sh": Process exited with status 1

sabe o que pode ser?

Alesson Viana · Answer 6 · Mon Jun 05 2023 20:56:26 GMT+0800 (China Standard Time)

Hey folks, any solution?
I'm facing the same problem faced by @cesarfior

César Augusto Fior · Answer 7 · Tue Jun 06 2023 09:42:03 GMT+0800 (China Standard Time)

Boa noite, esta falhando para baixar as dependências, parece ser um problema na key
oci_core_instance.["4"] (remote-exec): W: GPG error: https://packages.cloud.google.com/apt kubernetes-xenial InRelease: The following signatures couldn't be verified because the public key is not available: NO_PUBKEY B53DC80D13EDEF05
oci_core_instance.["4"] (remote-exec): E: The repository 'https://apt.kubernetes.io kubernetes-xenial InRelease' is not signed.
oci_core_instance._["4"] (remote-exec): 2023-06-06 01:26:42,119 - util.py[WARNING]: Running module apt-configure (<module 'cloudinit.config.cc_apt_configure' from '/usr/lib/python3/dist-packages/cloudinit/config/cc_apt_configure.py'>) failed

César Augusto Fior · Answer 8 · Wed Jun 07 2023 06:23:16 GMT+0800 (China Standard Time)

jpetazzo Boa noite, sabe como podemos resolver esse erro?

Daniel Lombardi · Answer 9 · Thu Jun 15 2023 21:02:23 GMT+0800 (China Standard Time)

Também estou com esse erro

Jérôme Petazzoni · Answer 10 · Wed Jul 05 2023 03:23:54 GMT+0800 (China Standard Time)

Hi everyone! It looks like an extra firewalling rule had been added to the Oracle images. I removed that rule (in commit 0a82500) and it looks like it solved it. Let me know if it works for you!

Olá pessoal! Parece que uma regra extra de firewall foi adicionada às imagens do Oracle. Eu removi essa regra (no commit 0a82500) e parece que resolveu. Me avise se funcionar para você!