Load balancer external-ip pending
jteng opened this issue · comments
I deploy the same deployment and expose it via the same kubectl expose deployment supervisor-deployment --type=LoadBalancer --name=supervisor-service command, in docker-desktop, a service is created successfully with external-ip assigned, but in kubernetes-admin@kind cluster, the service is created with external-ip pending forever.
here is the output from kubectl get svc on docker-desktop:
supervisor-service LoadBalancer 10.100.184.32 localhost 9090:31778/TCP 13s
while the kind cluster:
supervisor-service LoadBalancer 10.111.158.50 <pending> 9090:31054/TCP 16m
kind does not support type=LoadBalancer currently, load balancers are cloud provider specific. see #99 for some discussion around this.
thanks @BenTheElder
kind does not support type=LoadBalancer currently, load balancers are cloud provider specific. see #99 for some discussion around this.
@BenTheElder should we create a kind load balancer? seems a nice and funny project
@aojea I agree. It would be pretty interesting if we could test novel load balancers in our own desktop. Besides, some distributed machine learning infrastructure, e.g., kubeflow, require an external load balancer to work with.
You can deploy a loadbalancer with kind of course, but the integrated Kubernetes object is typically only supported "in the cloud" and Kubernetes does not ship one.
On docker for Mac in particular this is not quite so trivial.
Kubeflow should be able to work fine with something like a nodePort or an ingress, I will speak to them.
For others who stumble upon this (as I did), I've had nice success using Inlets to get a public IP for LoadBalancer services.
f you are not using GCE or EKS (you used kubeadm) you can add an externalIPs spec to your service YAML. You can use the IP associated with your node's primary interface such as eth0. You can then access the service externally, using the external IP of the node.
...
spec:
type: LoadBalancer
externalIPs:
- 192.168.0.10
On linux you can deploy metallb, but on mac / windows docker containers are not reachable from the host (only via a special port forwarding mechanism).
See also:
https://kind.sigs.k8s.io/docs/user/ingress/
https://kind.sigs.k8s.io/docs/user/resources/#how-to-use-kind-with-metalllb
https://docs.docker.com/docker-for-mac/networking/#i-cannot-ping-my-containers
docker/for-mac#2670
Docker Desktop on macOS users can use TunTap with MetalLB. Here's an example that feedback shows still works on macOS as of at least Catalina: https://github.com/AlmirKadric-Published/docker-tuntap-osx
I've also seen some interesting solutions using dnsmasq. K3s with k3d also ships with its own load balancer which can be adjusted at cluster creation time and multiple clusters are supported.
see #1961, which also brings up tuntap.
we're going to start by writing down the existing options.
The k3d "loadbalancer" implementation (1) is more or less the same thing KIND is doing is doing in HA mode. You still need to manually specify port mapping at cluster creation time, which is more or less analagous to https://kind.sigs.k8s.io/docs/user/configuration/#extra-port-mappings
Without something like tuntap there isn't a good way to have arbitrary mappings at runtime on mac / windows, since the containers are just not routable (only the port forwarding, which is some vpnkit magic specific to proxying ports).
On linux you can just deploy metallb and move on. Metallb CI uses kind. But then any developers on mac / windows are pretty SOL.
I found a good tutorial for getting EXTERNAL-IP to assign dynamically from a.pool using MetalLB on Mac. It's using TunTap and works on macOS 11. Because Linux is (and should be) the primary development platform, this gives as least some reprieve to those feeling the pain on macOS: https://www.thehumblelab.com/kind-and-metallb-on-mac/
I haven't had a chance to try this yet myself but it looks good! Previously #1961 (comment), also reached out to OP on twitter 😅
https://twitter.com/Codydearkland/status/1334946807202238464?s=19 🧵
I think at the very least this option needs to be referenced in our docs #1961