Support VPN only functionality on DigitalOcean managed k8s
spigell opened this issue · comments
Greetings! I use kilo only as a vpn server to inCluster resources, i.e. I have only one pod with kilo as deployment. On bare metal k3s cluster it works fine. But on DO managed k8s there are some troubles:
- nodes doesn't have a wireguard module. Trying to use boringtun.
- Even with boringtun interface do not in UP state and no config applied to it.
There is my deployment (also tried as DaemonSet WITH/OR HostNet+privileged):
apiVersion: apps/v1
kind: Deployment
metadata:
labels:
app.kubernetes.io/managed-by: pulumi
app.kubernetes.io/name: kilo
app.kubernetes.io/part-of: kilo
name: kilo
spec:
replicas: 1
selector:
matchLabels:
app.kubernetes.io/name: kilo
app.kubernetes.io/part-of: kilo
template:
metadata:
labels:
app.kubernetes.io/name: kilo
app.kubernetes.io/part-of: kilo
spec:
containers:
- args:
- kilo0
- --foreground
- --verbosity=debug
- --disable-drop-privileges=true
image: leonnicolas/boringtun
name: boringtun
securityContext:
privileged: true
volumeMounts:
- mountPath: /var/run/wireguard
name: wireguard
readOnly: false
- args:
- --kubeconfig=/etc/kubernetes/kubeconfig
- --hostname=$(NODE_NAME)
- --cni=false
- --log-level=all
- --port=51821
- --create-interface=false
- --topology-label=test.io/region
- --interface=kilo0
env:
- name: NODE_NAME
valueFrom:
fieldRef:
fieldPath: spec.nodeName
image: squat/kilo:latest
name: kilo
ports:
- containerPort: 1107
name: metrics
securityContext:
capabilities:
add:
- NET_ADMIN
- SYS_MODULE
privileged: false
volumeMounts:
- mountPath: /etc/kubernetes
name: kubeconfig
- mountPath: /scripts/
name: scripts
readOnly: true
- mountPath: /lib/modules
name: lib-modules
readOnly: true
- mountPath: /run/xtables.lock
name: xtables-lock
readOnly: false
- mountPath: /var/run/wireguard
name: wireguard
readOnly: false
- mountPath: /var/lib/kilo
name: kilo-dir
dnsPolicy: ClusterFirstWithHostNet
hostNetwork: false
initContainers:
- args:
- /scripts/init.sh
command:
- /bin/sh
env:
- name: NAMESPACE
valueFrom:
fieldRef:
fieldPath: metadata.namespace
image: squat/kilo
name: prepare-configs
volumeMounts:
- mountPath: /etc/kubernetes
name: kubeconfig
- mountPath: /scripts/
name: scripts
readOnly: true
- mountPath: /var/lib/kilo
name: kilo-dir
- mountPath: secrets
name: key
serviceAccountName: kilo
volumes:
- name: kilo-dir
- name: key
secret:
secretName: kilo-private-key
- hostPath:
path: /lib/modules
name: lib-modules
- hostPath:
path: /var/run/wireguard
name: wireguard
- name: kubeconfig
- configMap:
name: kilo-scripts
name: scripts
- hostPath:
path: /run/xtables.lock
type: FileOrCreate
name: xtables-lock
Starting logs:
boringtun Dec 08 00:19:33.796 INFO boringtun: BoringTun started successfully
boringtun at src/main.rs:186
boringtun
kilo {"caller":"mesh.go:143","component":"kilo","level":"debug","msg":"using 10.244.1.77/32 as the private IP address","ts":"2021-12-08T00:19:34.835604164Z"}
kilo {"caller":"mesh.go:154","component":"kilo","level":"debug","msg":"using 10.244.1.77/32 as the public IP address","ts":"2021-12-08T00:19:34.835705287Z"}
kilo {"caller":"main.go:274","msg":"Starting Kilo network mesh 'ee480dece4ceab3fd68b1f4a09e4e67da25003a6'.","ts":"2021-12-08T00:19:34.838064595Z"}
kilo {"caller":"mesh.go:277","component":"kilo","event":"add","level":"debug","msg":"syncing nodes","ts":"2021-12-08T00:19:35.045470243Z"}
kilo {"caller":"mesh.go:279","component":"kilo","event":"add","level":"debug","msg":"processing local node","node":{"Endpoint":null,"Key":"","NoInternalIP":false,"InternalIP":null,"LastSeen":0,"Leader":false,"Location":"","Name":"k8s-simple-uwlih","PersistentKeepalive":0,"Subnet":null,"WireGuardIP":null,"DiscoveredEndpoints":null,"AllowedLocationIPs":null,"Granularity":""},"ts":"2021-12-08T00:19:35.045566238Z"}
kilo {"caller":"mesh.go:396","component":"kilo","level":"debug","msg":"local node differs from backend","ts":"2021-12-08T00:19:35.046097141Z"}
kilo {"caller":"mesh.go:402","component":"kilo","level":"debug","msg":"successfully reconciled local node against backend","ts":"2021-12-08T00:19:35.056785065Z"}
kilo {"caller":"mesh.go:277","component":"kilo","event":"add","level":"debug","msg":"syncing nodes","ts":"2021-12-08T00:19:35.05690891Z"}
kilo {"caller":"mesh.go:288","component":"kilo","event":"add","in-mesh":false,"level":"debug","msg":"received non ready node","node":{"Endpoint":null,"Key":"","NoInternalIP":false,"InternalIP":null,"LastSeen":0,"Leader":false,"Location":"","Name":"k8s-simple-uwlik","PersistentKeepalive":0,"Subnet":null,"WireGuardIP":null,"DiscoveredEndpoints":null,"AllowedLocationIPs":null,"Granularity":""},"ts":"2021-12-08T00:19:35.057043385Z"}
kilo {"caller":"mesh.go:306","component":"kilo","event":"add","level":"info","node":{"Endpoint":null,"Key":"","NoInternalIP":false,"InternalIP":null,"LastSeen":0,"Leader":false,"Location":"","Name":"k8s-simple-uwlik","PersistentKeepalive":0,"Subnet":null,"WireGuardIP":null,"DiscoveredEndpoints":null,"AllowedLocationIPs":null,"Granularity":""},"ts":"2021-12-08T00:19:35.057125143Z"}
kilo {"caller":"mesh.go:277","component":"kilo","event":"update","level":"debug","msg":"syncing nodes","ts":"2021-12-08T00:19:35.058249307Z"}
kilo {"caller":"mesh.go:279","component":"kilo","event":"update","level":"debug","msg":"processing local node","node":{"Endpoint":{"DNS":"","IP":"10.244.1.77","Port":51821},"Key":"VHZxU1o0NEZlYlJEVWE3d1BTblVrbVk0ek40aTZXZDFReXBySndLMktuUT0=","NoInternalIP":false,"InternalIP":{"IP":"10.244.1.77","Mask":"/////w=="},"LastSeen":1638922775,"Leader":false,"Location":"","Name":"k8s-simple-uwlih","PersistentKeepalive":0,"Subnet":null,"WireGuardIP":null,"DiscoveredEndpoints":null,"AllowedLocationIPs":null,"Granularity":"location"},"ts":"2021-12-08T00:19:35.05835424Z"}
Inside the pod:
bash-5.0# ip l
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
2: tunl0@NONE: <NOARP,UP,LOWER_UP> mtu 1480 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000
link/ipip 0.0.0.0 brd 0.0.0.0
3: kilo0: <POINTOPOINT,MULTICAST,NOARP> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 500
link/none
16: eth0@if17: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default
link/ether d2:3f:6f:24:9e:0e brd ff:ff:ff:ff:ff:ff link-netnsid 0
bash-5.0# wg
interface: kilo0
listening port: 42557
There is no errors, pod works fine. If i will setup a interface correctly via ip
and wg set
commands it starts working. The DO managed k8s uses cillium as cni and I aware that there is no support fot it. Is there any change to make it working only as vpn gateway with support of CR Peer?