kubeip not work with GKE 1.12.6 single node?
remoe opened this issue · comments
I try to setup kubeip on a single node (one node pool with one node) cluster. After I apply the configuration the node goes in the "NotReady" state. The node still stay in this state after deleting the kubeip deployment.
@remoe Kubeip will not work if there is only one node
@avivl can you please explain a bit about why it wont work in a one node cluster. Is this a limitation of the current software and can be improved or an Architectural Decision that can not be fixed.
Basically my question is regarding how it handles dynamic situations in the Kubernetes cluster for example upgrading Kubernetes version, Autoscaling etc. Another question is does it work well with existing clusters with already assigned ips? I am currently testing it on a new cluster.
@yra-wtag KubeIP was designed for situations when you have multiple nodes in your GKE cluster but you want outgoing traffic to originate from a specific node or nodes. If your cluster has only one node, it means all your outgoing traffic will originate from this node and KubeIP will not be required. Make sense?
If your cluster has only one node, it means all your outgoing traffic will originate from this node and KubeIP will not be required. Make sense?
@spark2ignite
Well if you have auto-scaling then you can have 1 to as many you like nodes. These nodes are created and deleted automatically, so how does kubeip handle this situation?
The fact is the cluster does not have a fixed amount of nodes, it might have 1 to 5 nodes depending on the load of the service. So in our situation, KubeIP needs to assign static IP in a dynamic situation, when there is one node assign one ip from the IP list if there are 5 nodes assign 5 IP from the IP list. Is it possible? currently or after some code change?
also it seems there might be confusion the Architecture
kubeIP tries to solve this problem by assigning GKE nodes external IP addresses from a predefined list by continually watching the Kubernetes API for new/removed nodes and applying changes accordingly.
and your comment
KubeIP was designed for situations when you have multiple nodes in your GKE cluster but you want outgoing traffic to originate from a specific node or nodes.
This seems more like a proxy that all outgoing traffic automatically routed through these nodes.
However kubeIP is a CloudNAT alternative according to this https://blog.doit-intl.com/kubeip-automatically-assign-external-static-ips-to-your-gke-nodes-for-easier-whitelisting-without-2068eb9c14cd
so which one is it.
@eranchetz can you please look into this?
@eranchetz, I don't have this environment anymore. But I wonder how this should work, because in the docs it has:
We recommend that KUBEIP_NODEPOOL should NOT be the same as KUBEIP_SELF_NODEPOOL
So, when you have a one node nodepool you can't have two nodepools.
@remoe In my case I set the same node pool for both even though it is not recommended (which makes perfect sense). But it works nonetheless.
@eranchetz , I've also tried this in the past. But I had still the same issue. Which GKE version you have used?
@remoe I now see I used 1.11.8-gke.6,
I'll create a new cluster and try again with 1.12.6.
@remoe OK, I was able to reproduce the issue with 1.12.6-gke.10, but, after a few seconds the node changed back to the "Ready" state, Any chance you can reproduce the issue and send me the details, I will try to follow your instructions.
@eranchetz i have been able to reproduce the issue on 1.12.7-gke.10 ; As you have said previously the node indeed went back to Ready state in a previous cluster i tried. However i have tried twice in 1.12.7-gke.10 did not change IP after half an hour.
This is how the process worked
- I followed the instructions on a g1-small (1 vCPU, 1.7 GB memory) node, regarding ip address, service account etc.
- As every other node it starts with a default External Ip
- Then i deployed using
kubectl apply -f deploy/.The External IP was removed, but the correct ip was never assigned
It got stuck in the unknown state indefinitely.
Let me know if i can help in any way.
NB: So i moved to the other way. Created another node, deployed kubeip there it worked properly. Seems like there might be a design issue in the system as @avivl told. I think the Readme should say that KubeIP does not work on single node cluster.
@yra-wtag , Ok, I will try to reproduce it myself with the same parameters and update.
thanks 👍
@yra-wtag I was still unable to reproduce with your parameters.
In case you reproduce it again, can you please share the output of:
kubectl logs `kubectl get pods -n kube-system -oname -lapp=kubeip` -n kube-systemThanks again
@yra-wtag I was still unable to reproduce with your parameters.
In case you reproduce it again, can you please share the output of:kubectl logs `kubectl get pods -n kube-system -oname -lapp=kubeip` -n kube-systemThanks again
@eranchetz thanks for the reply, i tried this too, unfortunately there is no logs. I think kubernetes lost connection to this node.
@yra-wtag , can you please try with this PR #56 ? It was the first time that kube-ip works for me ;)
KUBEIP_LABELKEY: "kubeip"
KUBEIP_LABELVALUE: "gke1"
KUBEIP_NODEPOOL: "pool1"
KUBEIP_FORCEASSIGNMENT: "true"
KUBEIP_ADDITIONALNODEPOOLS: ""
KUBEIP_TICKER: "5"
KUBEIP_ALLNODEPOOLS: "false"
KUBEIP_SELF_NODEPOOL: "small"
@yra-wtag , can you please try with this PR #56 ? It was the first time that kube-ip works for me ;)
KUBEIP_LABELKEY: "kubeip" KUBEIP_LABELVALUE: "gke1" KUBEIP_NODEPOOL: "pool1" KUBEIP_FORCEASSIGNMENT: "true" KUBEIP_ADDITIONALNODEPOOLS: "" KUBEIP_TICKER: "5" KUBEIP_ALLNODEPOOLS: "false" KUBEIP_SELF_NODEPOOL: "small"
Yeah we are already using that.