Networking issues with Raspbian Lite / RPi4
alexellis opened this issue · comments
Describe the bug
A clear and concise description of what the bug is.
I flashed Raspbian Lite for my Raspberry Pi 4 (4GB) and updated / upgraded packages with apt:
Linux k4s-3 4.19.58-v7l+ rancher/k3s#1245 SMP Fri Jul 12 17:31:45 BST 2019 armv7l GNU/Linux
I did this on 3 separate RPis and ran the curl / sh script to install k3s as a server on each. I then tried to access Traefik on port 80 with the Node's IP (using my laptop). It did not respond.
If I use curl from on the RPi it does respond with 404 not found.
I then deployed OpenFaaS with the following:
git clone https://github.com/openfaas/faas-netes && cd faas-netes && sudo kubectl apply -f namespaces.yml,yaml_armhf
The pods which rely on networking failed to start, namely the ones which talk to Prometheus on start-up, or the ones which talk to NATS on start-up.
sudo kubectl get pods -A
Additional context
Add any other context about the problem here.
This worked well for me every time I tried it previously. The only thing which I think is different is having run sudo apt upgrade. I did not run the firmware update command.
Is DNS inside the pods working?
I can power the device up and run a quick command if you have something handy for me to test with?
@ibuildthecloud DNS isn't working and IP connectivity isn't working either.
CoreDNS appears to be started.
lsmod
Module Size Used by
ipt_REJECT 16384 0
nf_reject_ipv4 16384 1 ipt_REJECT
xt_multiport 16384 1
veth 24576 0
xt_nat 16384 11
xt_tcpudp 16384 37
vxlan 49152 0
ip6_udp_tunnel 16384 1 vxlan
udp_tunnel 16384 1 vxlan
nft_chain_nat_ipv6 16384 4
nf_nat_ipv6 20480 1 nft_chain_nat_ipv6
xt_comment 16384 45
iptable_filter 16384 1
xt_mark 16384 7
iptable_nat 16384 2
ip_vs_sh 16384 0
ip_vs_wrr 16384 0
ip_vs_rr 16384 0
ip_vs 143360 6 ip_vs_wrr,ip_vs_rr,ip_vs_sh
ipt_MASQUERADE 16384 7
nf_conntrack_netlink 40960 0
nft_counter 16384 15
nft_chain_nat_ipv4 16384 4
nf_nat_ipv4 16384 3 nft_chain_nat_ipv4,ipt_MASQUERADE,iptable_nat
xt_addrtype 16384 5
nft_compat 20480 4
nf_tables 122880 46 nft_chain_nat_ipv6,nft_chain_nat_ipv4,nft_compat,nft_counter
nfnetlink 16384 4 nft_compat,nf_conntrack_netlink,nf_tables
xt_conntrack 16384 7
nf_nat 36864 3 xt_nat,nf_nat_ipv6,nf_nat_ipv4
nf_conntrack 135168 8 ip_vs,xt_nat,ipt_MASQUERADE,nf_conntrack_netlink,nf_nat_ipv6,xt_conntrack,nf_nat_ipv4,nf_nat
nf_defrag_ipv6 20480 1 nf_conntrack
nf_defrag_ipv4 16384 1 nf_conntrack
br_netfilter 24576 0
bridge 135168 1 br_netfilter
overlay 106496 11
bnep 20480 2
hci_uart 40960 1
btbcm 16384 1 hci_uart
serdev 20480 1 hci_uart
bluetooth 389120 24 hci_uart,bnep,btbcm
ecdh_generic 28672 1 bluetooth
8021q 32768 0
garp 16384 1 8021q
stp 16384 2 garp,bridge
llc 16384 3 garp,bridge,stp
brcmfmac 311296 0
brcmutil 16384 1 brcmfmac
sha256_generic 20480 0
vc4 172032 0
cfg80211 614400 1 brcmfmac
drm_kms_helper 184320 1 vc4
bcm2835_codec 36864 0
rfkill 28672 6 bluetooth,cfg80211
raspberrypi_hwmon 16384 0
bcm2835_v4l2 45056 0
hwmon 16384 1 raspberrypi_hwmon
v4l2_mem2mem 24576 1 bcm2835_codec
bcm2835_mmal_vchiq 32768 2 bcm2835_codec,bcm2835_v4l2
v4l2_common 16384 1 bcm2835_v4l2
videobuf2_dma_contig 20480 1 bcm2835_codec
videobuf2_vmalloc 16384 1 bcm2835_v4l2
videobuf2_memops 16384 2 videobuf2_dma_contig,videobuf2_vmalloc
v3d 61440 0
videobuf2_v4l2 24576 3 bcm2835_codec,bcm2835_v4l2,v4l2_mem2mem
snd_soc_core 192512 1 vc4
videobuf2_common 45056 4 bcm2835_codec,bcm2835_v4l2,v4l2_mem2mem,videobuf2_v4l2
gpu_sched 28672 1 v3d
snd_compress 20480 1 snd_soc_core
videodev 200704 6 bcm2835_codec,v4l2_common,videobuf2_common,bcm2835_v4l2,v4l2_mem2mem,videobuf2_v4l2
snd_pcm_dmaengine 16384 1 snd_soc_core
media 36864 2 videodev,v4l2_mem2mem
syscopyarea 16384 1 drm_kms_helper
vc_sm_cma 36864 1 bcm2835_mmal_vchiq
sysfillrect 16384 1 drm_kms_helper
drm 438272 5 v3d,vc4,gpu_sched,drm_kms_helper
sysimgblt 16384 1 drm_kms_helper
fb_sys_fops 16384 1 drm_kms_helper
drm_panel_orientation_quirks 16384 1 drm
snd_bcm2835 24576 1
snd_pcm 102400 4 vc4,snd_pcm_dmaengine,snd_bcm2835,snd_soc_core
snd_timer 32768 1 snd_pcm
snd 73728 7 snd_compress,snd_timer,snd_bcm2835,snd_soc_core,snd_pcm
argon_mem 16384 0
uio_pdrv_genirq 16384 0
fixed 16384 0
uio 20480 1 uio_pdrv_genirq
ip_tables 24576 2 iptable_filter,iptable_nat
x_tables 32768 12 xt_comment,xt_multiport,ipt_REJECT,xt_nat,ip_tables,nft_compat,iptable_filter,xt_mark,xt_tcpudp,ipt_MASQUERADE,xt_addrtype,xt_conntrack
ipv6 450560 249 nf_nat_ipv6,bridge
Alex
Thanks for filing this issue @alexellis!
I am trying to replicate on a Raspberry Pi 2 (armv7) but am having a hard time doing so. I deployed OpenFaas using the commands you provided:
root@k3s-base:~/faas-netes# kubectl get all -A
NAMESPACE NAME READY STATUS RESTARTS AGE
kube-system pod/coredns-b7464766c-cckf5 1/1 Running 0 10m
kube-system pod/helm-install-traefik-w2d7g 0/1 Completed 0 10m
kube-system pod/svclb-traefik-xm5c4 2/2 Running 0 8m26s
kube-system pod/traefik-56688c4464-mhv7x 1/1 Running 0 8m25s
openfaas pod/alertmanager-757cc474bc-6rqfw 1/1 Running 0 7m11s
openfaas pod/faas-idler-59dfd85f6c-vgnhm 1/1 Running 2 7m12s
openfaas pod/gateway-597f6578bc-7v9ch 2/2 Running 0 7m12s
openfaas pod/nats-d4c9d8d95-bwfwf 1/1 Running 0 7m11s
openfaas pod/prometheus-68d68d7466-xwclk 1/1 Running 0 7m9s
openfaas pod/queue-worker-df9d5749c-zgfrh 1/1 Running 0 7m9s
NAMESPACE NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
default service/kubernetes ClusterIP 10.43.0.1 <none> 443/TCP 10m
kube-system service/kube-dns ClusterIP 10.43.0.10 <none> 53/UDP,53/TCP,9153/TCP 10m
kube-system service/traefik LoadBalancer 10.43.48.154 10.20.3.70 80:31121/TCP,443:31253/TCP 8m26s
openfaas service/alertmanager ClusterIP 10.43.96.238 <none> 9093/TCP 7m12s
openfaas service/gateway ClusterIP 10.43.61.241 <none> 8080/TCP 7m12s
openfaas service/gateway-external NodePort 10.43.106.66 <none> 8080:31112/TCP 7m12s
openfaas service/nats ClusterIP 10.43.107.135 <none> 4222/TCP 7m11s
openfaas service/prometheus ClusterIP 10.43.121.9 <none> 9090/TCP 7m10s
NAMESPACE NAME DESIRED CURRENT READY UP-TO-DATE AVAILABLE NODE SELECTOR AGE
kube-system daemonset.apps/svclb-traefik 1 1 1 1 1 <none> 8m26s
NAMESPACE NAME READY UP-TO-DATE AVAILABLE AGE
kube-system deployment.apps/coredns 1/1 1 1 10m
kube-system deployment.apps/traefik 1/1 1 1 8m26s
openfaas deployment.apps/alertmanager 1/1 1 1 7m12s
openfaas deployment.apps/faas-idler 1/1 1 1 7m12s
openfaas deployment.apps/gateway 1/1 1 1 7m12s
openfaas deployment.apps/nats 1/1 1 1 7m11s
openfaas deployment.apps/prometheus 1/1 1 1 7m11s
openfaas deployment.apps/queue-worker 1/1 1 1 7m10s
NAMESPACE NAME DESIRED CURRENT READY AGE
kube-system replicaset.apps/coredns-b7464766c 1 1 1 10m
kube-system replicaset.apps/traefik-56688c4464 1 1 1 8m26s
openfaas replicaset.apps/alertmanager-757cc474bc 1 1 1 7m12s
openfaas replicaset.apps/faas-idler-59dfd85f6c 1 1 1 7m12s
openfaas replicaset.apps/gateway-597f6578bc 1 1 1 7m12s
openfaas replicaset.apps/nats-d4c9d8d95 1 1 1 7m11s
openfaas replicaset.apps/prometheus-68d68d7466 1 1 1 7m10s
openfaas replicaset.apps/queue-worker-df9d5749c 1 1 1 7m9s
NAMESPACE NAME COMPLETIONS DURATION AGE
kube-system job.batch/helm-install-traefik 1/1 101s 10mThe logs for CoreDNS:
root@k3s-base:~/faas-netes# kubectl logs -n kube-system deployment.apps/coredns
.:53
2019-08-06T18:39:47.494Z [INFO] CoreDNS-1.3.0
2019-08-06T18:39:47.495Z [INFO] linux/arm, go1.11.4, c8f0e94
CoreDNS-1.3.0
linux/arm, go1.11.4, c8f0e94
2019-08-06T18:39:47.495Z [INFO] plugin/reload: Running configuration MD5 = ef347efee19aa82f09972f89f92da1cfFollowing the instructions for testing DNS at https://kubernetes.io/docs/tasks/administer-cluster/dns-debugging-resolution/
root@k3s-base:~/faas-netes# kubectl apply -f https://k8s.io/examples/admin/dns/busybox.yaml
pod/busybox createdAnd resolving the OpenFaas service:
root@k3s-base:~/faas-netes# kubectl exec -ti busybox -- nslookup nats.openfaas.svc.cluster.local
Server: 10.43.0.10
Address 1: 10.43.0.10 kube-dns.kube-system.svc.cluster.local
Name: nats.openfaas.svc.cluster.local
Address 1: 10.43.107.135 nats.openfaas.svc.cluster.localI am then able to access the portal through port 31112:
Thanks for spending some time on it. There are at least two things you are doing differently.
- I used RPi4
- I ran
sudo apt upgrade -qy
I did mention the upgrade step in the original post. Upgrading your RPi3 packages may break, or this may only break with RPi4.
Alex
Take a note of my uname -a output, can you try to compare it to yours before and after running sudo apt update && sudo apt upgrade --qy?
Linux k4s-3 4.19.58-v7l+ #1245 SMP Fri Jul 12 17:31:45 BST 2019 armv7l GNU/Linux
Ok, original uname -a:
Linux k3s-base 4.14.79-v7+ #1159 SMP Sun Nov 4 17:50:20 GMT 2018 armv7l GNU/Linux
After upgrading:
Linux k3s-base 4.19.57-v7+ #1244 SMP Thu Jul 4 18:45:25 BST 2019 armv7l GNU/Linux
Pods:
root@k3s-base:~# kubectl get pods -A
NAMESPACE NAME READY STATUS RESTARTS AGE
default pod/busybox 1/1 Running 3 157m
kube-system pod/coredns-b7464766c-cckf5 1/1 Running 1 169m
kube-system pod/helm-install-traefik-w2d7g 0/1 Completed 0 169m
kube-system pod/svclb-traefik-xm5c4 2/2 Running 2 167m
kube-system pod/traefik-56688c4464-mhv7x 1/1 Running 1 167m
openfaas pod/alertmanager-757cc474bc-6rqfw 1/1 Running 1 166m
openfaas pod/faas-idler-59dfd85f6c-vgnhm 1/1 Running 5 166m
openfaas pod/gateway-597f6578bc-7v9ch 2/2 Running 3 166m
openfaas pod/nats-d4c9d8d95-bwfwf 1/1 Running 1 166m
openfaas pod/prometheus-68d68d7466-xwclk 1/1 Running 1 166m
openfaas pod/queue-worker-df9d5749c-zgfrh 1/1 Running 2 166m
Retrying with a fresh install produced the same (working) result. Some of the pods entered CrashLoopBackOff state until everything stabilized, which took about 4m30s.
Sorry I don't have an RPi4 to test with, but if you could provide more info, like the CoreDNS logs and service resolve test that would be awesome.
It may be that the RPi just needs more time to bring everything up, and eventually all of your pods will stay in a running state.
Please see these two examples.
Working RPi4:
- only has k3s / git
- didn't get
sudo apt upgrade - Only pod that restarted was
faas-idlertwice
RPi4 with k3s which is not working:
- has k3s and latest
dockerpackage - did get
sudo apt upgrade
Still crashing after 22m, due to no DNS, no IP connectivity.
These are the modules loaded:
Module Size Used by
ipt_REJECT 16384 3
nf_reject_ipv4 16384 1 ipt_REJECT
xt_multiport 16384 1
xt_nat 16384 14
xt_tcpudp 16384 49
veth 24576 0
nft_chain_nat_ipv6 16384 4
nf_nat_ipv6 20480 1 nft_chain_nat_ipv6
vxlan 49152 0
ip6_udp_tunnel 16384 1 vxlan
udp_tunnel 16384 1 vxlan
xt_comment 16384 54
xt_mark 16384 7
iptable_nat 16384 2
iptable_filter 16384 1
ip_vs_sh 16384 0
ip_vs_wrr 16384 0
ip_vs_rr 16384 0
ip_vs 143360 6 ip_vs_wrr,ip_vs_rr,ip_vs_sh
ipt_MASQUERADE 16384 7
nf_conntrack_netlink 40960 0
nft_counter 16384 15
nft_chain_nat_ipv4 16384 4
nf_nat_ipv4 16384 3 nft_chain_nat_ipv4,ipt_MASQUERADE,iptable_nat
xt_addrtype 16384 6
nft_compat 20480 4
nf_tables 122880 46 nft_chain_nat_ipv6,nft_chain_nat_ipv4,nft_compat,nft_counter
nfnetlink 16384 4 nft_compat,nf_conntrack_netlink,nf_tables
xt_conntrack 16384 7
nf_nat 36864 3 xt_nat,nf_nat_ipv6,nf_nat_ipv4
nf_conntrack 135168 8 ip_vs,xt_nat,ipt_MASQUERADE,nf_conntrack_netlink,nf_nat_ipv6,xt_conntrack,nf_nat_ipv4,nf_nat
nf_defrag_ipv6 20480 1 nf_conntrack
nf_defrag_ipv4 16384 1 nf_conntrack
br_netfilter 24576 0
bridge 135168 1 br_netfilter
overlay 106496 21
bnep 20480 2
hci_uart 40960 1
btbcm 16384 1 hci_uart
serdev 20480 1 hci_uart
bluetooth 389120 24 hci_uart,bnep,btbcm
ecdh_generic 28672 1 bluetooth
8021q 32768 0
garp 16384 1 8021q
stp 16384 2 garp,bridge
llc 16384 3 garp,bridge,stp
brcmfmac 311296 0
brcmutil 16384 1 brcmfmac
sha256_generic 20480 0
cfg80211 614400 1 brcmfmac
vc4 172032 0
rfkill 28672 6 bluetooth,cfg80211
v3d 61440 0
drm_kms_helper 184320 1 vc4
gpu_sched 28672 1 v3d
bcm2835_codec 36864 0
drm 438272 5 v3d,vc4,gpu_sched,drm_kms_helper
raspberrypi_hwmon 16384 0
drm_panel_orientation_quirks 16384 1 drm
bcm2835_v4l2 45056 0
v4l2_mem2mem 24576 1 bcm2835_codec
hwmon 16384 1 raspberrypi_hwmon
bcm2835_mmal_vchiq 32768 2 bcm2835_codec,bcm2835_v4l2
videobuf2_dma_contig 20480 1 bcm2835_codec
v4l2_common 16384 1 bcm2835_v4l2
snd_soc_core 192512 1 vc4
videobuf2_vmalloc 16384 1 bcm2835_v4l2
videobuf2_memops 16384 2 videobuf2_dma_contig,videobuf2_vmalloc
videobuf2_v4l2 24576 3 bcm2835_codec,bcm2835_v4l2,v4l2_mem2mem
videobuf2_common 45056 4 bcm2835_codec,bcm2835_v4l2,v4l2_mem2mem,videobuf2_v4l2
snd_bcm2835 24576 1
snd_compress 20480 1 snd_soc_core
snd_pcm_dmaengine 16384 1 snd_soc_core
videodev 200704 6 bcm2835_codec,v4l2_common,videobuf2_common,bcm2835_v4l2,v4l2_mem2mem,videobuf2_v4l2
media 36864 2 videodev,v4l2_mem2mem
vc_sm_cma 36864 1 bcm2835_mmal_vchiq
snd_pcm 102400 4 vc4,snd_pcm_dmaengine,snd_bcm2835,snd_soc_core
syscopyarea 16384 1 drm_kms_helper
sysfillrect 16384 1 drm_kms_helper
sysimgblt 16384 1 drm_kms_helper
snd_timer 32768 1 snd_pcm
fb_sys_fops 16384 1 drm_kms_helper
snd 73728 7 snd_compress,snd_timer,snd_bcm2835,snd_soc_core,snd_pcm
argon_mem 16384 0
fixed 16384 0
uio_pdrv_genirq 16384 0
uio 20480 1 uio_pdrv_genirq
ip_tables 24576 2 iptable_filter,iptable_nat
x_tables 32768 12 xt_comment,xt_multiport,ipt_REJECT,xt_nat,ip_tables,nft_compat,iptable_filter,xt_mark,xt_tcpudp,ipt_MASQUERADE,xt_addrtype,xt_conntrack
ipv6 450560 397 nf_nat_ipv6,bridge
vs. on the working unit:
Module Size Used by
nft_chain_nat_ipv6 16384 4
nf_nat_ipv6 20480 1 nft_chain_nat_ipv6
nf_tables 122880 1 nft_chain_nat_ipv6
xt_multiport 16384 1
nf_conntrack_netlink 40960 0
nfnetlink 16384 2 nf_conntrack_netlink,nf_tables
veth 24576 0
vxlan 49152 0
ip6_udp_tunnel 16384 1 vxlan
udp_tunnel 16384 1 vxlan
xt_nat 16384 16
xt_addrtype 16384 3
ipt_REJECT 16384 0
nf_reject_ipv4 16384 1 ipt_REJECT
xt_tcpudp 16384 52
ipt_MASQUERADE 16384 6
xt_conntrack 16384 6
xt_comment 16384 55
iptable_filter 16384 1
xt_mark 16384 7
iptable_nat 16384 2
nf_nat_ipv4 16384 2 ipt_MASQUERADE,iptable_nat
nf_nat 36864 3 xt_nat,nf_nat_ipv6,nf_nat_ipv4
ip_vs_sh 16384 0
ip_vs_wrr 16384 0
ip_vs_rr 16384 0
ip_vs 143360 6 ip_vs_wrr,ip_vs_rr,ip_vs_sh
nf_conntrack 135168 8 ip_vs,xt_nat,ipt_MASQUERADE,nf_conntrack_netlink,nf_nat_ipv6,xt_conntrack,nf_nat_ipv4,nf_nat
nf_defrag_ipv6 20480 1 nf_conntrack
nf_defrag_ipv4 16384 1 nf_conntrack
overlay 106496 22
br_netfilter 24576 0
bridge 135168 1 br_netfilter
bnep 20480 2
hci_uart 40960 1
btbcm 16384 1 hci_uart
serdev 20480 1 hci_uart
bluetooth 389120 24 hci_uart,bnep,btbcm
ecdh_generic 28672 1 bluetooth
8021q 32768 0
garp 16384 1 8021q
stp 16384 2 garp,bridge
llc 16384 3 garp,bridge,stp
brcmfmac 311296 0
brcmutil 16384 1 brcmfmac
sha256_generic 20480 0
cfg80211 614400 1 brcmfmac
rfkill 28672 6 bluetooth,cfg80211
vc4 172032 0
v3d 61440 0
drm_kms_helper 184320 1 vc4
gpu_sched 28672 1 v3d
raspberrypi_hwmon 16384 0
hwmon 16384 1 raspberrypi_hwmon
snd_soc_core 192512 1 vc4
snd_compress 20480 1 snd_soc_core
drm 438272 5 v3d,vc4,gpu_sched,drm_kms_helper
snd_pcm_dmaengine 16384 1 snd_soc_core
bcm2835_v4l2 45056 0
snd_bcm2835 24576 1
bcm2835_codec 36864 0
syscopyarea 16384 1 drm_kms_helper
snd_pcm 102400 4 vc4,snd_pcm_dmaengine,snd_bcm2835,snd_soc_core
sysfillrect 16384 1 drm_kms_helper
v4l2_mem2mem 24576 1 bcm2835_codec
snd_timer 32768 1 snd_pcm
sysimgblt 16384 1 drm_kms_helper
drm_panel_orientation_quirks 16384 1 drm
bcm2835_mmal_vchiq 32768 2 bcm2835_codec,bcm2835_v4l2
snd 73728 7 snd_compress,snd_timer,snd_bcm2835,snd_soc_core,snd_pcm
fb_sys_fops 16384 1 drm_kms_helper
v4l2_common 16384 1 bcm2835_v4l2
videobuf2_dma_contig 20480 1 bcm2835_codec
videobuf2_vmalloc 16384 1 bcm2835_v4l2
videobuf2_memops 16384 2 videobuf2_dma_contig,videobuf2_vmalloc
videobuf2_v4l2 24576 3 bcm2835_codec,bcm2835_v4l2,v4l2_mem2mem
videobuf2_common 45056 4 bcm2835_codec,bcm2835_v4l2,v4l2_mem2mem,videobuf2_v4l2
videodev 200704 6 bcm2835_codec,v4l2_common,videobuf2_common,bcm2835_v4l2,v4l2_mem2mem,videobuf2_v4l2
media 36864 2 videodev,v4l2_mem2mem
argon_mem 16384 0
uio_pdrv_genirq 16384 0
fixed 16384 0
uio 20480 1 uio_pdrv_genirq
ip_tables 24576 2 iptable_filter,iptable_nat
x_tables 32768 11 xt_comment,xt_multiport,ipt_REJECT,xt_nat,ip_tables,iptable_filter,xt_mark,xt_tcpudp,ipt_MASQUERADE,xt_addrtype,xt_conntrack
ipv6 450560 482 nf_nat_ipv6,bridge
Guess what happened after I uninstalled docker-ce? ^
I suspect the containerd/runc version that ships with Docker may be breaking k3s.
Thanks for the info @alexellis!
If installed via curl script k3s will prepend its binary path (ie /var/lib/rancher/k3s/data/.../bin) and use the runc in that location. The k3s or containerd logs may contain more information. iptables may be breaking also, doing a diff of iptables-save before & after might help.
I have the same issue with a Raspberry Pi 4 cluster dns and ip does not work after installing docker
Uninstalling docker restored network I suppose some iptables issue
My work-around was to remove docker completely, then remove k3s and add it back in again.
It would be ideal if both of these tools could co-exist, so that we can do builds on cluster servers or agents.
Got docker and k3s working together, issue appears that k3s uses iptables-legacy and docker is using iptables that in Buster is really iptables-nft, from issue kubernetes/kubernetes#71305 having both tables active is recipe for disaster
Running:
sudo iptables -F
sudo update-alternatives --set iptables /usr/sbin/iptables-legacy
and a reboot fixed and allows both to run in the cluster or worker
Thank you for debugging and finding the resolution @hectoregm !
I think we should maybe prefer the system iptables to our bundled version, similar to what we do with mount https://github.com/rancher/k3s/blob/e6817c3809aa4d0d247a213f68cff881eb948a54/scripts/download#L15
At the moment I think we should just add a warning to the install script if using an older version of iptables since this will be difficult to address for all projects.
The upstream iptables-nft binary probably needs to fallback to legacy binary or kernel modules if nft is not available.
I had exactly the same issue this weekend. New docker messes up iptables so using solo containerd is the solution. Thanks guys!
No movement on this issue since November. @erikwilson what do you think we should do here?
@rancher-max - based on changes made since the issue was last looked at, can you give this a go and test this out again?
I tried this a few different ways using 1.17 1.18 and 1.19 k3s and all seem to be working now. I installed docker version 19.03.13 on my rpi4 and tried with kernel versions 4 and 5. When using v1.17.4 k3s I was able to replicate this issue by running:
kubectl run -it --rm --restart=Never dnsutils --image=gcr.io/kubernetes-e2e-test-images/dnsutils:1.3 sh and then ping -c 1 google.com in the resulting container. It would get 100% packet loss and not reach outside, whereas now it is successful.
I believe this is fixed and am closing it out. If future issues arise related to this feel free to reopen or create a new issue.