Support VPN only functionality on DigitalOcean managed k8s

Question

Support VPN only functionality on DigitalOcean managed k8s

spigell opened this issue 2 years ago · comments

Greetings! I use kilo only as a vpn server to inCluster resources, i.e. I have only one pod with kilo as deployment. On bare metal k3s cluster it works fine. But on DO managed k8s there are some troubles:

nodes doesn't have a wireguard module. Trying to use boringtun.
Even with boringtun interface do not in UP state and no config applied to it.

There is my deployment (also tried as DaemonSet WITH/OR HostNet+privileged):

apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app.kubernetes.io/managed-by: pulumi
    app.kubernetes.io/name: kilo
    app.kubernetes.io/part-of: kilo
  name: kilo
spec:
  replicas: 1
  selector:
    matchLabels:
      app.kubernetes.io/name: kilo
      app.kubernetes.io/part-of: kilo
  template:
    metadata:
      labels:
        app.kubernetes.io/name: kilo
        app.kubernetes.io/part-of: kilo
    spec:
      containers:
      - args:
        - kilo0
        - --foreground
        - --verbosity=debug
        - --disable-drop-privileges=true
        image: leonnicolas/boringtun
        name: boringtun
        securityContext:
          privileged: true
        volumeMounts:
        - mountPath: /var/run/wireguard
          name: wireguard
          readOnly: false
      - args:
        - --kubeconfig=/etc/kubernetes/kubeconfig
        - --hostname=$(NODE_NAME)
        - --cni=false
        - --log-level=all
        - --port=51821
        - --create-interface=false
        - --topology-label=test.io/region
        - --interface=kilo0
        env:
        - name: NODE_NAME
          valueFrom:
            fieldRef:
              fieldPath: spec.nodeName
        image: squat/kilo:latest
        name: kilo
        ports:
        - containerPort: 1107
          name: metrics
        securityContext:
          capabilities:
            add:
            - NET_ADMIN
            - SYS_MODULE
          privileged: false
        volumeMounts:
        - mountPath: /etc/kubernetes
          name: kubeconfig
        - mountPath: /scripts/
          name: scripts
          readOnly: true
        - mountPath: /lib/modules
          name: lib-modules
          readOnly: true
        - mountPath: /run/xtables.lock
          name: xtables-lock
          readOnly: false
        - mountPath: /var/run/wireguard
          name: wireguard
          readOnly: false
        - mountPath: /var/lib/kilo
          name: kilo-dir
      dnsPolicy: ClusterFirstWithHostNet
      hostNetwork: false
      initContainers:
      - args:
        - /scripts/init.sh
        command:
        - /bin/sh
        env:
        - name: NAMESPACE
          valueFrom:
            fieldRef:
              fieldPath: metadata.namespace
        image: squat/kilo
        name: prepare-configs
        volumeMounts:
        - mountPath: /etc/kubernetes
          name: kubeconfig
        - mountPath: /scripts/
          name: scripts
          readOnly: true
        - mountPath: /var/lib/kilo
          name: kilo-dir
        - mountPath: secrets
          name: key
      serviceAccountName: kilo
      volumes:
      - name: kilo-dir
      - name: key
        secret:
          secretName: kilo-private-key
      - hostPath:
          path: /lib/modules
        name: lib-modules
      - hostPath:
          path: /var/run/wireguard
        name: wireguard
      - name: kubeconfig
      - configMap:
          name: kilo-scripts
        name: scripts
      - hostPath:
          path: /run/xtables.lock
          type: FileOrCreate
        name: xtables-lock

Starting logs:

boringtun   Dec 08 00:19:33.796  INFO boringtun: BoringTun started successfully
boringtun     at src/main.rs:186
boringtun 
kilo {"caller":"mesh.go:143","component":"kilo","level":"debug","msg":"using 10.244.1.77/32 as the private IP address","ts":"2021-12-08T00:19:34.835604164Z"}
kilo {"caller":"mesh.go:154","component":"kilo","level":"debug","msg":"using 10.244.1.77/32 as the public IP address","ts":"2021-12-08T00:19:34.835705287Z"}
kilo {"caller":"main.go:274","msg":"Starting Kilo network mesh 'ee480dece4ceab3fd68b1f4a09e4e67da25003a6'.","ts":"2021-12-08T00:19:34.838064595Z"}
kilo {"caller":"mesh.go:277","component":"kilo","event":"add","level":"debug","msg":"syncing nodes","ts":"2021-12-08T00:19:35.045470243Z"}
kilo {"caller":"mesh.go:279","component":"kilo","event":"add","level":"debug","msg":"processing local node","node":{"Endpoint":null,"Key":"","NoInternalIP":false,"InternalIP":null,"LastSeen":0,"Leader":false,"Location":"","Name":"k8s-simple-uwlih","PersistentKeepalive":0,"Subnet":null,"WireGuardIP":null,"DiscoveredEndpoints":null,"AllowedLocationIPs":null,"Granularity":""},"ts":"2021-12-08T00:19:35.045566238Z"}
kilo {"caller":"mesh.go:396","component":"kilo","level":"debug","msg":"local node differs from backend","ts":"2021-12-08T00:19:35.046097141Z"}
kilo {"caller":"mesh.go:402","component":"kilo","level":"debug","msg":"successfully reconciled local node against backend","ts":"2021-12-08T00:19:35.056785065Z"}
kilo {"caller":"mesh.go:277","component":"kilo","event":"add","level":"debug","msg":"syncing nodes","ts":"2021-12-08T00:19:35.05690891Z"}
kilo {"caller":"mesh.go:288","component":"kilo","event":"add","in-mesh":false,"level":"debug","msg":"received non ready node","node":{"Endpoint":null,"Key":"","NoInternalIP":false,"InternalIP":null,"LastSeen":0,"Leader":false,"Location":"","Name":"k8s-simple-uwlik","PersistentKeepalive":0,"Subnet":null,"WireGuardIP":null,"DiscoveredEndpoints":null,"AllowedLocationIPs":null,"Granularity":""},"ts":"2021-12-08T00:19:35.057043385Z"}
kilo {"caller":"mesh.go:306","component":"kilo","event":"add","level":"info","node":{"Endpoint":null,"Key":"","NoInternalIP":false,"InternalIP":null,"LastSeen":0,"Leader":false,"Location":"","Name":"k8s-simple-uwlik","PersistentKeepalive":0,"Subnet":null,"WireGuardIP":null,"DiscoveredEndpoints":null,"AllowedLocationIPs":null,"Granularity":""},"ts":"2021-12-08T00:19:35.057125143Z"}
kilo {"caller":"mesh.go:277","component":"kilo","event":"update","level":"debug","msg":"syncing nodes","ts":"2021-12-08T00:19:35.058249307Z"}
kilo {"caller":"mesh.go:279","component":"kilo","event":"update","level":"debug","msg":"processing local node","node":{"Endpoint":{"DNS":"","IP":"10.244.1.77","Port":51821},"Key":"VHZxU1o0NEZlYlJEVWE3d1BTblVrbVk0ek40aTZXZDFReXBySndLMktuUT0=","NoInternalIP":false,"InternalIP":{"IP":"10.244.1.77","Mask":"/////w=="},"LastSeen":1638922775,"Leader":false,"Location":"","Name":"k8s-simple-uwlih","PersistentKeepalive":0,"Subnet":null,"WireGuardIP":null,"DiscoveredEndpoints":null,"AllowedLocationIPs":null,"Granularity":"location"},"ts":"2021-12-08T00:19:35.05835424Z"}

Inside the pod:

bash-5.0# ip l
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
2: tunl0@NONE: <NOARP,UP,LOWER_UP> mtu 1480 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000
    link/ipip 0.0.0.0 brd 0.0.0.0
3: kilo0: <POINTOPOINT,MULTICAST,NOARP> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 500
    link/none 
16: eth0@if17: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default 
    link/ether d2:3f:6f:24:9e:0e brd ff:ff:ff:ff:ff:ff link-netnsid 0

bash-5.0# wg
interface: kilo0
  listening port: 42557

There is no errors, pod works fine. If i will setup a interface correctly via ip and wg set commands it starts working. The DO managed k8s uses cillium as cni and I aware that there is no support fot it. Is there any change to make it working only as vpn gateway with support of CR Peer?