No access to service-cidr with netstack
rlex opened this issue · comments
Description
k3s v1.22.3+k3s1
containerd://1.5.7-k3s2
cilium 1.10.0 (if that matters). Kube-proxy replacement enabled with eBPF dataplane.
gvisor installed and working.
However, DNS resolution fails, because gvisor without host-network fails to access 10.43.0.0/16 network which is assigned service-cidr. Network otherwise works (i was able to curl https://1.1.1.1, for example).
Works fine with host-network, but i assume this is overkill for just dns access?
Happy to help with debugging if you can point me to right direction.
Steps to reproduce
containerd config:
[plugins.opt]
path = "/var/lib/rancher/k3s/agent/containerd"
[plugins.cri]
stream_server_address = "127.0.0.1"
stream_server_port = "10010"
enable_selinux = false
sandbox_image = "rancher/mirrored-pause:3.1"
[plugins.cri.containerd]
snapshotter = "overlayfs"
disable_snapshot_annotations = true
[plugins.cri.containerd.runtimes.runc]
runtime_type = "io.containerd.runc.v2"
[plugins.cri.containerd.runtimes.runsc]
runtime_type = "io.containerd.runsc.v1"
[plugins.cri.containerd.runtimes.runsc-hostnetwork]
runtime_type = "io.containerd.runsc.v1"
[plugins.cri.containerd.runtimes.runsc.options]
TypeUrl = "io.containerd.runsc.v1.options"
ConfigPath = "/var/lib/rancher/k3s/agent/etc/containerd/runsc.toml"
[plugins.cri.containerd.runtimes.runsc-hostnetwork.options]
TypeUrl = "io.containerd.runsc.v1.options"
ConfigPath = "/var/lib/rancher/k3s/agent/etc/containerd/runsc-hostnetwork.toml"
(I have two runtimeclasses with hostnetwork and without)
/var/lib/rancher/k3s/agent/etc/containerd/runsc.toml is empty atm
runsc version
runsc version release-20211129.0
spec: 1.0.2docker version (if using docker)
No response
uname
Linux node-4 5.11.0-43-generic #47~20.04.2-Ubuntu SMP Mon Dec 13 11:06:56 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
kubectl (if using Kubernetes)
No response
repo state (if built from source)
No response
runsc debug logs (if available)
No response
routing table on gvisor with netstack (using sample nginx pod):
root@gvisor:/# cat /proc/net/route
Iface Destination Gateway Flags RefCnt Use Metric Mask MTU Window IRTT
eth0 F802000A 00000000 0001 0 0 0 FFFFFFFF 0 0 0
eth0 00000000 F802000A 0003 0 0 0 00000000 0 0 0
root@gvisor:/# route
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
10.0.2.248 0.0.0.0 255.255.255.255 U 0 0 0 eth0
default 10.0.2.248 0.0.0.0 UG 0 0 0 eth0
with hostnetwork:
root@gvisor-hostnetwork:/# cat /proc/net/route
Iface Destination Gateway Flags RefCnt Use Metric Mask MTU Window IRTT
eth0 00000000 EC06000A 0003 0 0 0 00000000 0 0 0
eth0 EC06000A 00000000 0001 0 0 0 FFFFFFFF 0 0 0
eth0 3206000A 00000000 0001 0 0 0 FFFFFFFF 0 0 0
root@gvisor-hostnetwork:/# route
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
default 10.0.6.236 0.0.0.0 UG 0 0 0 eth0
10.0.6.236 0.0.0.0 255.255.255.255 U 0 0 0 eth0
10.0.6.50 0.0.0.0 255.255.255.255 U 0 0 0 eth0
Do you have a minimal repro? I am not sure I follow what DNS and the address in this comment have to do with each other
.
"However, DNS resolution fails, because gvisor without host-network fails to access 10.43.0.0/16 network which is assigned"
DNS (coredns in my example) is deployed as an in-cluster service with ip 10.43.0.10, which is part of service-cidr (k3s default is 10.43.0.0/16)
Other IPs in this range can't be reached with default gvisor config (non-hostnetwork one), for example default "kubernetes" service located on 10.43.0.1 which is kubernetes-api.
As soon as i enable host-network in runsc config, both DNS and kubernetes API starts working.
Sorry I dropped the ball on this. Got busy with other things. It would be great if you could setup a demo cluster/project and give me access so that I can see what's going on.
@hbhasker yeah it's possible, i can spin up some small vm somewhere. How can i contact you, slack / etc?
Please drop me a note at bhaskerh at Google dot com.
A friendly reminder that this issue had no activity for 120 days.
This issue has been closed due to lack of activity.