service暴露的端口使用公网ip无法访问
Vortexxxx opened this issue · comments
做到部署node节点那一步，暂时master和node都在一台上，跟据教程创建了一个niginx的service试一下集群是否可用：
Pod状态：
[root@node-2 /]$kubectl get pods
NAME READY STATUS RESTARTS AGE
nginx-ff994b94c-tv472 1/1 Running 0 5m
nginx-ff994b94c-z74hl 1/1 Running 0 5m
service状态：
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
example-service NodePort 10.30.92.37 <none> 80:30109/TCP 6m
kubernetes ClusterIP 10.30.0.1 <none> 443/TCP 11m
-------------
[root@node-2 /]$kubectl describe svc example-service
Name: example-service
Namespace: default
Labels: run=load-balancer-example
Annotations: <none>
Selector: run=load-balancer-example
Type: NodePort
IP: 10.30.92.37
Port: <unset> 80/TCP
TargetPort: 80/TCP
NodePort: <unset> 30109/TCP
Endpoints: 172.30.29.2:80,172.30.29.3:80
Session Affinity: None
External Traffic Policy: Cluster
Events: <none>
使用curl 10.30.92.37可以得到nginx的响应，
[root@node-2 /]$curl 10.30.92.37
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
<style>
body {
width: 35em;
margin: 0 auto;
font-family: Tahoma, Verdana, Arial, sans-serif;
}
</style>
</head>
<body>
<h1>Welcome to nginx!</h1>
<p>If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.</p>
<p>For online documentation and support please refer to
<a href="http://nginx.org/">nginx.org</a>.<br/>
Commercial support is available at
<a href="http://nginx.com/">nginx.com</a>.</p>
<p><em>Thank you for using nginx.</em></p>
</body>
</html>
但是使用从公网访问curl 公网ip:30109则无响应，浏览器也没有报错，用curl开debug模式：
$ time curl 198.44.242.180:32708 -v
* Rebuilt URL to: 198.44.242.180:30109/
* Trying 198.44.242.180...
* TCP_NODELAY set
* Connected to 198.44.242.180 (198.44.242.180) port 30109 (#0)
> GET / HTTP/1.1
> Host: 198.44.242.180:30109
> User-Agent: curl/7.54.0
> Accept: */*
>
* Empty reply from server
* Connection #0 to host 198.44.242.180 left intact
curl: (52) Empty reply from server
curl 198.44.242.180:30109 -v 0.01s user 0.01s system 0% cpu 2:02.37 total
检查了api-server，kube-proxy, kubelet都是正常运行的状态：
[root@node-2 /]$kubectl get componentstatuses
NAME STATUS MESSAGE ERROR
scheduler Healthy ok
controller-manager Healthy ok
etcd-0 Healthy {"health": "true"}
etcd-1 Healthy {"health": "true"}
@Vortexxxx 你需要确保 公网ip:30109 与 节点ip:30109 有映射关系，并且 enable 防火墙。如果你是使用公有云，可以考虑 loadbalancer type 的 Service。
抱歉， 我刚刚接触这个东西，有一些概念可能没明白
我是使用一个国外的小vps做测试的，应该是没有node balance之类的服务的
你需要确保 公网ip:30109 与 节点ip:30109 有映射关系,并且 enable 防火墙
我不太明白这句话中的公网ip和节点ip有映射关系，我上个ifconfig:
docker0 Link encap:Ethernet HWaddr 02:42:8c:41:ae:64
inet addr:172.30.60.1 Bcast:0.0.0.0 Mask:255.255.255.0
inet6 addr: fe80::42:8cff:fe41:ae64/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1450 Metric:1
RX packets:23940 errors:0 dropped:0 overruns:0 frame:0
TX packets:24219 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:4583563 (4.5 MB) TX bytes:2135299 (2.1 MB)
eno1 Link encap:Ethernet HWaddr 0c:c4:7a:c3:4a:52
inet addr:192.168.1.27 Bcast:192.168.1.255 Mask:255.255.255.0
inet6 addr: fe80::ec4:7aff:fec3:4a52/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:176954539 errors:0 dropped:0 overruns:0 frame:0
TX packets:127122871 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:17322891642 (17.3 GB) TX bytes:11573237120 (11.5 GB)
Memory:df920000-df93ffff
eno2 Link encap:Ethernet HWaddr 0c:c4:7a:c3:4a:53
inet addr:198.44.242.180 Bcast:198.44.242.255 Mask:255.255.255.0
inet6 addr: fe80::ec4:7aff:fec3:4a53/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:1805477994 errors:0 dropped:0 overruns:0 frame:0
TX packets:5111116 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:111425503256 (111.4 GB) TX bytes:331179782 (331.1 MB)
Memory:df900000-df91ffff
flannel.1 Link encap:Ethernet HWaddr 4e:e4:30:fa:11:f6
inet addr:172.30.60.0 Bcast:0.0.0.0 Mask:255.255.255.255
inet6 addr: fe80::4ce4:30ff:fefa:11f6/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1450 Metric:1
RX packets:0 errors:0 dropped:0 overruns:0 frame:0
TX packets:0 errors:0 dropped:8 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:0 (0.0 B) TX bytes:0 (0.0 B)
lo Link encap:Local Loopback
inet addr:127.0.0.1 Mask:255.0.0.0
inet6 addr: ::1/128 Scope:Host
UP LOOPBACK RUNNING MTU:65536 Metric:1
RX packets:2377317 errors:0 dropped:0 overruns:0 frame:0
TX packets:2377317 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1
RX bytes:571589242 (571.5 MB) TX bytes:571589242 (571.5 MB)
198.44.242.180 是我的公网ip，192.168.1.27是内网ip（同时应该是你所指的节点ip）
通过netstat -nl 查看,32708(后面重新部署了一次service，端口变了)是监听公网的，但是就是无法访问
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address Foreign Address State
tcp 0 0 192.168.1.27:10255 0.0.0.0:* LISTEN
tcp 0 0 192.168.1.27:8080 0.0.0.0:* LISTEN
tcp 0 0 0.0.0.0:8822 0.0.0.0:* LISTEN
tcp 0 0 192.168.1.27:4194 0.0.0.0:* LISTEN
tcp 0 0 127.0.0.1:10248 0.0.0.0:* LISTEN
tcp 0 0 127.0.0.1:10249 0.0.0.0:* LISTEN
tcp 0 0 192.168.1.27:10250 0.0.0.0:* LISTEN
tcp 0 0 192.168.1.27:6443 0.0.0.0:* LISTEN
tcp 0 0 127.0.0.1:10251 0.0.0.0:* LISTEN
tcp 0 0 192.168.1.27:2379 0.0.0.0:* LISTEN
tcp 0 0 127.0.0.1:2379 0.0.0.0:* LISTEN
tcp 0 0 127.0.0.1:10252 0.0.0.0:* LISTEN
tcp 0 0 192.168.1.27:2380 0.0.0.0:* LISTEN
tcp6 0 0 :::10256 :::* LISTEN
tcp6 0 0 :::32708 :::* LISTEN
udp 0 0 0.0.0.0:8472 0.0.0.0:*
操作系统是Ubuntu16.04，iptables确认是可用的
可以先在机器内部试试 192.168.1.27:30109 是不是正常，如果正常但 198.44.242.180:30109 不正常的话，多半是VM的安全组禁止掉了端口的访问。
service信息如下：
[root@fanya_mysql /]$kubectl get svc
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
example-service NodePort 10.254.130.28 <none> 80:32708/TCP 11h
kubernetes ClusterIP 10.254.0.1 <none> 443/TCP 14h
在服务器上
curl 192.168.1.27:32708 #内网ip
curl 198.44.242.180:32708 #公网ip
curl 10.254.130.28 #CLUSTER-IP
以上三个请求都能得到正确的结果
但是从我本机就是无法访问
附上iptables的信息以供参考：
# Generated by iptables-save v1.6.0 on Mon Nov 6 11:31:38 2017
*nat
:PREROUTING ACCEPT [0:0]
:INPUT ACCEPT [0:0]
:OUTPUT ACCEPT [1:60]
:POSTROUTING ACCEPT [1:60]
:DOCKER - [0:0]
:KUBE-MARK-DROP - [0:0]
:KUBE-MARK-MASQ - [0:0]
:KUBE-NODEPORTS - [0:0]
:KUBE-POSTROUTING - [0:0]
:KUBE-SEP-4BUG4J4CVKOHQ2R6 - [0:0]
:KUBE-SEP-ISOJDKHZW44FGYF5 - [0:0]
:KUBE-SEP-JPKWSAPUEOPRRN6W - [0:0]
:KUBE-SEP-YU2NNQNFRQGXRUD7 - [0:0]
:KUBE-SERVICES - [0:0]
:KUBE-SVC-BR4KARPIGKMRMN3E - [0:0]
:KUBE-SVC-NPX46M4PTMTKRN6Y - [0:0]
:KUBE-SVC-XGLOHA7QRQ3V22RZ - [0:0]
-A PREROUTING -m comment --comment "kubernetes service portals" -j KUBE-SERVICES
-A PREROUTING -m addrtype --dst-type LOCAL -j DOCKER
-A OUTPUT -m comment --comment "kubernetes service portals" -j KUBE-SERVICES
-A OUTPUT ! -d 127.0.0.0/8 -m addrtype --dst-type LOCAL -j DOCKER
-A POSTROUTING -m comment --comment "kubernetes postrouting rules" -j KUBE-POSTROUTING
-A POSTROUTING -s 172.30.60.0/24 ! -o docker0 -j MASQUERADE
-A DOCKER -i docker0 -j RETURN
-A KUBE-MARK-DROP -j MARK --set-xmark 0x8000/0x8000
-A KUBE-MARK-MASQ -j MARK --set-xmark 0x4000/0x4000
-A KUBE-NODEPORTS -p tcp -m comment --comment "default/example-service:" -m tcp --dport 32708 -j KUBE-MARK-MASQ
-A KUBE-NODEPORTS -p tcp -m comment --comment "default/example-service:" -m tcp --dport 32708 -j KUBE-SVC-BR4KARPIGKMRMN3E
-A KUBE-POSTROUTING -m comment --comment "kubernetes service traffic requiring SNAT" -m mark --mark 0x4000/0x4000 -j MASQUERADE
-A KUBE-SEP-4BUG4J4CVKOHQ2R6 -s 172.30.60.3/32 -m comment --comment "default/example-service:" -j KUBE-MARK-MASQ
-A KUBE-SEP-4BUG4J4CVKOHQ2R6 -p tcp -m comment --comment "default/example-service:" -m tcp -j DNAT --to-destination 172.30.60.3:80
-A KUBE-SEP-ISOJDKHZW44FGYF5 -s 172.30.60.4/32 -m comment --comment "kube-system/kubernetes-dashboard:" -j KUBE-MARK-MASQ
-A KUBE-SEP-ISOJDKHZW44FGYF5 -p tcp -m comment --comment "kube-system/kubernetes-dashboard:" -m tcp -j DNAT --to-destination 172.30.60.4:9090
-A KUBE-SEP-JPKWSAPUEOPRRN6W -s 192.168.1.27/32 -m comment --comment "default/kubernetes:https" -j KUBE-MARK-MASQ
-A KUBE-SEP-JPKWSAPUEOPRRN6W -p tcp -m comment --comment "default/kubernetes:https" -m recent --set --name KUBE-SEP-JPKWSAPUEOPRRN6W --mask 255.255.255.255 --rsource -m tcp -j DNAT --to-destination 192.168.1.27:6443
-A KUBE-SEP-YU2NNQNFRQGXRUD7 -s 172.30.60.2/32 -m comment --comment "default/example-service:" -j KUBE-MARK-MASQ
-A KUBE-SEP-YU2NNQNFRQGXRUD7 -p tcp -m comment --comment "default/example-service:" -m tcp -j DNAT --to-destination 172.30.60.2:80
-A KUBE-SERVICES ! -s 10.254.0.0/16 -d 10.254.130.28/32 -p tcp -m comment --comment "default/example-service: cluster IP" -m tcp --dport 80 -j KUBE-MARK-MASQ
-A KUBE-SERVICES -d 10.254.130.28/32 -p tcp -m comment --comment "default/example-service: cluster IP" -m tcp --dport 80 -j KUBE-SVC-BR4KARPIGKMRMN3E
-A KUBE-SERVICES ! -s 10.254.0.0/16 -d 10.254.0.1/32 -p tcp -m comment --comment "default/kubernetes:https cluster IP" -m tcp --dport 443 -j KUBE-MARK-MASQ
-A KUBE-SERVICES -d 10.254.0.1/32 -p tcp -m comment --comment "default/kubernetes:https cluster IP" -m tcp --dport 443 -j KUBE-SVC-NPX46M4PTMTKRN6Y
-A KUBE-SERVICES ! -s 10.254.0.0/16 -d 10.254.53.98/32 -p tcp -m comment --comment "kube-system/kubernetes-dashboard: cluster IP" -m tcp --dport 80 -j KUBE-MARK-MASQ
-A KUBE-SERVICES -d 10.254.53.98/32 -p tcp -m comment --comment "kube-system/kubernetes-dashboard: cluster IP" -m tcp --dport 80 -j KUBE-SVC-XGLOHA7QRQ3V22RZ
-A KUBE-SERVICES -m comment --comment "kubernetes service nodeports; NOTE: this must be the last rule in this chain" -m addrtype --dst-type LOCAL -j KUBE-NODEPORTS
-A KUBE-SVC-BR4KARPIGKMRMN3E -m comment --comment "default/example-service:" -m statistic --mode random --probability 0.50000000000 -j KUBE-SEP-YU2NNQNFRQGXRUD7
-A KUBE-SVC-BR4KARPIGKMRMN3E -m comment --comment "default/example-service:" -j KUBE-SEP-4BUG4J4CVKOHQ2R6
-A KUBE-SVC-NPX46M4PTMTKRN6Y -m comment --comment "default/kubernetes:https" -m recent --rcheck --seconds 10800 --reap --name KUBE-SEP-JPKWSAPUEOPRRN6W --mask 255.255.255.255 --rsource -j KUBE-SEP-JPKWSAPUEOPRRN6W
-A KUBE-SVC-NPX46M4PTMTKRN6Y -m comment --comment "default/kubernetes:https" -j KUBE-SEP-JPKWSAPUEOPRRN6W
-A KUBE-SVC-XGLOHA7QRQ3V22RZ -m comment --comment "kube-system/kubernetes-dashboard:" -j KUBE-SEP-ISOJDKHZW44FGYF5
COMMIT
# Completed on Mon Nov 6 11:31:38 2017
# Generated by iptables-save v1.6.0 on Mon Nov 6 11:31:38 2017
*filter
:INPUT ACCEPT [363:84124]
:FORWARD DROP [0:0]
:OUTPUT ACCEPT [350:82783]
:DOCKER - [0:0]
:DOCKER-ISOLATION - [0:0]
:KUBE-FIREWALL - [0:0]
:KUBE-SERVICES - [0:0]
-A INPUT -m comment --comment "kubernetes service portals" -j KUBE-SERVICES
-A INPUT -j KUBE-FIREWALL
-A FORWARD -j DOCKER-ISOLATION
-A FORWARD -o docker0 -j DOCKER
-A FORWARD -o docker0 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A FORWARD -i docker0 ! -o docker0 -j ACCEPT
-A FORWARD -i docker0 -o docker0 -j ACCEPT
-A OUTPUT -m comment --comment "kubernetes service portals" -j KUBE-SERVICES
-A OUTPUT -j KUBE-FIREWALL
-A DOCKER-ISOLATION -j RETURN
-A KUBE-FIREWALL -m comment --comment "kubernetes firewall for dropping marked packets" -m mark --mark 0x8000/0x8000 -j DROP
COMMIT
# Completed on Mon Nov 6 11:31:38 2017
嗯 看起来是VPS的安全组限制了32708端口的访问，可以到VPS Portal上检查看看。
我和主机商确认了一下，并没有外部的防火墙，于是我做了一个小测试，停掉svc，然后使用docker run -d -p 32708:80 nginx来运行一个容器：
[root@fanya_mysql /]$kubectl delete svc example-service
service "example-service" deleted
[root@fanya_mysql /]$docker run -d -p 32708:80 nginx
ee0f7d84e35d5c2e7310697e9f99da2fedfd29855713a33f5f8f9cbac823dc08
http://198.44.242.180:32708/这样是正常访问的，我重新创建了一个svc，还是无法访问，相关信息如下，所以应该还是k8s的问题，如果需要我补充任何信息请回复
[root@fanya_mysql /]$ kubectl expose deployment nginx --type=NodePort --name=example-service
service "example-service" exposed
[root@fanya_mysql /]$kubectl describe svc example-service
Name: example-service
Namespace: default
Labels: run=load-balancer-example
Annotations: <none>
Selector: run=load-balancer-example
Type: NodePort
IP: 10.254.170.200
Port: <unset> 80/TCP
TargetPort: 80/TCP
NodePort: <unset> 32675/TCP
Endpoints: 172.30.60.2:80,172.30.60.3:80
Session Affinity: None
External Traffic Policy: Cluster
Events: <none>
试试 iptables -P FORWARD ACCEPT
居然可以了，感谢！！！
虽然我知道这条命令是开启iptables的转发，但是我不太明白为什么执行这个之后k8s就正常工作了， 我没有特意的去动过iptables的设置，都是由程序操作的
能指教一下吗
这是新版本docker导致的问题，默认情况下这个是DROP
非常感谢！！！
那我再多观察一下这个问题，安装几个不同版本的试试
问题解决了我先把这个issue关掉了
试试
iptables -P FORWARD ACCEPT
thank you very much
@Vortexxxx 你需要确保
公网ip:30109与节点ip:30109有映射关系，并且 enable 防火墙。如果你是使用公有云，可以考虑 loadbalancer type 的 Service。
请教下怎么映射公网ip和节点ip？