Connectivity issues on cluster created by Rancher
fabioroger opened this issue · comments
Have you tested this in a non-gke environment?
Once my client connects to server, I can't access anything other than the openvpn pod.
Communication between client and the openvpn pod seems fine, but can't connect from client to anything else (in kubernetes cluster or not)
I tried changing OVPN_NETWORK
and also played with OVPN_DEFROUTE
, but of no avail. Also took a look on debug logs but no obvious errors show up.
Any ideas?
Initialization logs:
Fri Aug 24 18:03:59 2018 Running 'openvpn --config /etc/openvpn/openvpn.conf --push route 10.43.0.0 255.255.0.0 --push route 10.42.0.0 255.255.0.0 --client-config-dir /etc/openvpn/ccd --crl-verify /etc/openvpn/crl/crl.pem --status /etc/openvpn/status/server.status --status-version 2 '
Fri Aug 24 18:03:59 2018 OpenVPN 2.4.4 x86_64-alpine-linux-musl [SSL (OpenSSL)] [LZO] [LZ4] [EPOLL] [MH/PKTINFO] [AEAD] built on Nov 9 2017
Fri Aug 24 18:03:59 2018 library versions: LibreSSL 2.6.3, LZO 2.10
Fri Aug 24 18:03:59 2018 Diffie-Hellman initialized with 2048 bit key
Fri Aug 24 18:03:59 2018 Outgoing Control Channel Authentication: Using 160 bit message hash 'SHA1' for HMAC authentication
Fri Aug 24 18:03:59 2018 Incoming Control Channel Authentication: Using 160 bit message hash 'SHA1' for HMAC authentication
Fri Aug 24 18:03:59 2018 TUN/TAP device tun0 opened
Fri Aug 24 18:03:59 2018 TUN/TAP TX queue length set to 100
Fri Aug 24 18:03:59 2018 do_ifconfig, tt->did_ifconfig_ipv6_setup=0
Fri Aug 24 18:03:59 2018 /sbin/ip link set dev tun0 up mtu 1500
Fri Aug 24 18:03:59 2018 /sbin/ip addr add dev tun0 10.140.0.1/24 broadcast 10.140.0.255
Fri Aug 24 18:03:59 2018 Routing 10.42.0.36:20080 to 10.140.0.5:8000 (fabio)
Fri Aug 24 18:03:59 2018 Could not determine IPv4/IPv6 protocol. Using AF_INET
Fri Aug 24 18:03:59 2018 Socket Buffers: R=[87380->87380] S=[16384->16384]
Fri Aug 24 18:03:59 2018 Listening for incoming TCP connection on [AF_INET][undef]:1194
Fri Aug 24 18:03:59 2018 TCPv4_SERVER link local (bound): [AF_INET][undef]:1194
Fri Aug 24 18:03:59 2018 TCPv4_SERVER link remote: [AF_UNSPEC]
Fri Aug 24 18:03:59 2018 GID set to nogroup
Fri Aug 24 18:03:59 2018 UID set to nobody
Fri Aug 24 18:03:59 2018 MULTI: multi_init called, r=256 v=256
Fri Aug 24 18:03:59 2018 IFCONFIG POOL: base=10.140.0.2 size=252, ipv6=0
Fri Aug 24 18:03:59 2018 MULTI: TCP INIT maxclients=1024 maxevents=1028
Fri Aug 24 18:03:59 2018 Initialization Sequence Completed
Client connecting logs:
Fri Aug 24 18:05:18 2018 TCP connection established with [AF_INET]X.X.X.X:54972
Fri Aug 24 18:05:19 2018 X.X.X.X:54972 TLS: Initial packet from [AF_INET]X.X.X.X:54972, sid=e96080a4 ca7d61d1
Fri Aug 24 18:05:19 2018 X.X.X.X:54972 VERIFY OK: depth=1, CN=XXXX
Fri Aug 24 18:05:19 2018 X.X.X.X:54972 VERIFY OK: depth=0, CN=fabio
Fri Aug 24 18:05:19 2018 X.X.X.X:54972 peer info: IV_VER=2.4.6
Fri Aug 24 18:05:19 2018 X.X.X.X:54972 peer info: IV_PLAT=mac
Fri Aug 24 18:05:19 2018 X.X.X.X:54972 peer info: IV_PROTO=2
Fri Aug 24 18:05:19 2018 X.X.X.X:54972 peer info: IV_NCP=2
Fri Aug 24 18:05:19 2018 X.X.X.X:54972 peer info: IV_LZ4=1
Fri Aug 24 18:05:19 2018 X.X.X.X:54972 peer info: IV_LZ4v2=1
Fri Aug 24 18:05:19 2018 X.X.X.X:54972 peer info: IV_LZO=1
Fri Aug 24 18:05:19 2018 X.X.X.X:54972 peer info: IV_COMP_STUB=1
Fri Aug 24 18:05:19 2018 X.X.X.X:54972 peer info: IV_COMP_STUBv2=1
Fri Aug 24 18:05:19 2018 X.X.X.X:54972 peer info: IV_TCPNL=1
Fri Aug 24 18:05:19 2018 X.X.X.X:54972 peer info: IV_GUI_VER="net.tunnelblick.tunnelblick_5080_3.7.6a__build_5080)"
Fri Aug 24 18:05:19 2018 X.X.X.X:54972 Control Channel: TLSv1.2, cipher TLSv1/SSLv3 ECDHE-RSA-AES128-GCM-SHA256, 2048 bit RSA
Fri Aug 24 18:05:19 2018 X.X.X.X:54972 [fabio] Peer Connection Initiated with [AF_INET]X.X.X.X:54972
Fri Aug 24 18:05:19 2018 fabio/X.X.X.X:54972 OPTIONS IMPORT: reading client specific options from: /etc/openvpn/ccd/fabio
Fri Aug 24 18:05:19 2018 fabio/X.X.X.X:54972 MULTI: Learn: 10.140.0.5 -> fabio/X.X.X.X:54972
Fri Aug 24 18:05:19 2018 fabio/X.X.X.X:54972 MULTI: primary virtual IP for fabio/X.X.X.X:54972: 10.140.0.5
Fri Aug 24 18:05:20 2018 fabio/X.X.X.X:54972 PUSH: Received control message: 'PUSH_REQUEST'
Fri Aug 24 18:05:20 2018 fabio/X.X.X.X:54972 SENT CONTROL [fabio]: 'PUSH_REPLY,block-outside-dns,dhcp-option DOMAIN svc.cluster.local,dhcp-option DNS 10.43.0.10,route 10.43.0.0 255.255.0.0,route 10.42.0.0 255.255.0.0,route-gateway 10.140.0.1,topology subnet,ping 10,ping-restart 60,ifconfig 10.140.0.5 255.255.255.0,peer-id 0,cipher AES-256-GCM' (status=1)
Fri Aug 24 18:05:20 2018 fabio/X.X.X.X:54972 Data Channel: using negotiated cipher 'AES-256-GCM'
Fri Aug 24 18:05:20 2018 fabio/X.X.X.X:54972 Outgoing Data Channel: Cipher 'AES-256-GCM' initialized with 256 bit key
Fri Aug 24 18:05:20 2018 fabio/X.X.X.X:54972 Incoming Data Channel: Cipher 'AES-256-GCM' initialized with 256 bit key
netstat -rn:
Kernel IP routing table
Destination Gateway Genmask Flags MSS Window irtt Iface
0.0.0.0 169.254.1.1 0.0.0.0 UG 0 0 0 eth0
10.140.0.0 0.0.0.0 255.255.255.0 U 0 0 0 tun0
169.254.1.1 0.0.0.0 255.255.255.255 UH 0 0 0 eth0
iptables-save:
# Generated by iptables-save v1.6.1 on Fri Aug 24 18:09:38 2018
*nat
:PREROUTING ACCEPT [129:9276]
:INPUT ACCEPT [1:60]
:OUTPUT ACCEPT [0:0]
:POSTROUTING ACCEPT [0:0]
:KUBEOPENVPNPORTFORWARD - [0:0]
-A PREROUTING -j KUBEOPENVPNPORTFORWARD
-A POSTROUTING -s 10.140.0.0/24 -d 10.43.0.0/16 -o eth0 -j SNAT --to-source 10.42.0.36
-A POSTROUTING -s 10.140.0.0/24 -d 10.42.0.0/16 -o eth0 -j SNAT --to-source 10.42.0.36
-A KUBEOPENVPNPORTFORWARD -d 10.42.0.36/32 -p tcp -m tcp --dport 20080 -j DNAT --to-destination 10.140.0.5:8000
COMMIT
# Completed on Fri Aug 24 18:09:38 2018
Check the following:
- Did you configure the correct pod CIDR?
- Did you configure the correct service CIDR?
- Are there other things in your cluster restricting network access, such as
NetworkPolicies
?
How are you checking connectivity?
First, thank you very much for your reply.
So,
For pods CIDR:
$ kubectl cluster-info dump | grep -i cidr
"PodCIDR": "10.42.0.0/24",
"subnet": "usePodCidr"
For services CIDR, since all start with 10.43
and assumed their CIDR to be 10.43.0.0/16
.
As for NetworkPolicies
, I'm not at all familiar with them, but:
$ kubectl get NetworkPolicy --all-namespaces
No resources found.
And I'm checking connectivity with netcat
. Calls like this just hang forever:
$ nc -v 10.42.0.10 53
against all known services/pod ips and ports. The only ports I can reach are the ones in openvpn
own pod.
Also, for sanity check, I just redid the steps against a Google Cluster cluster and everything works perfectly.
Again thank you for your help. Any other ideas would be greatly appreciated.
Found the problem!
net.ipv4.ip_forward
was 0
for openvpn container.
Running this on the node did the trick:
$ nsenter -t $(docker inspect --format '{{.State.Pid}}' $CONTAINER_NAME) -n sysctl -w net.ipv4.ip_forward=1