Do you use MetalLB? Tell us!
danderson opened this issue · comments
This is not an issue so much as a lightweight way of gathering information on who is using MetalLB. This is mostly to satisfy our curiosity, but might also help us decide how to evolve the project.
So, if you use MetalLB for something, please chime in here and tell us more!
We're not using it yet, but our K8S clusters are 100% metallic, running in our data centers and on customer premises. We're going to be trying this out in the new year.
Our deployment would in most cases have this peer directly with our routers. We're a mostly Cisco shop. Which BGP implementations have you tested with?
So far, I've only tested with OSS BGP implementations, mainly BIRD. I used to have some Juniper and Cisco hardware that could speak BGP, but I got rid of them :(.
With that said, currently MetalLB speaks a very conventional, "has existed for ever" dialect of BGP, so it should work fine. I foresee two possible issues interacting with Cisco's implementation:
- Some old Cisco devices use an unusual encoding for capability advertisements in the OPEN message. AFAICT, they stopped doing this in the late 1990s, so it shouldn't be a problem. If it is a problem, it's a trivial patch to fix it.
- Cisco may refuse to speak to a BGPv4 speaker that uses the original route encoding for IPv4 (which the RFCs say is still 100% correct), and instead requires all route advertisements to use the MP-BGP encoding. I'm in the middle of making MetalLB speak MP-BGP for dual-stack support anyway, so hopefully by the time you test MetalLB this should not be a problem.
If you hit any interop problems, please file bugs! Ideally with a pcap that captures the initial exchange of BGP OPEN messages and a few UPDATE messages, so that I can add regression tests.
A colleague brought my attention to this project recently. We're running bare metal k8s on bgp enabled hosts and pods so this looks very interesting. Unfortunately we are using 32-bit AS numbers in our setup so we can't test it right away but it looks promising!
@xnaveira MetalLB supports 32-bit AS numbers! It's the one BGP extension that's implemented in v0.1.0. v0.2.0 will also support MP-BGP, so if you use Quagga or Cisco devices, you'll want to wait for 0.2.0.
That is awesome @danderson , i'll give it a try asap then.
I saw a post in reddit.com about MetalLB and this is what I had been looking for a long time for my local K8S on Mac. Took about one day following the tutorial and experiment, finally I have it up and running. Now I can deploy service in k8s with LoadBalancer without "pending". Awesome.
We have successfully deployed a patched version of MetalLB on a staging K8S cluster at @naitways, peered with JunOS hardware appliances. Awesome! :)
\o/ What patching did you need to interface with JunOS? I'm going to spin up a virtual JunOS later today and do interop testing, but it sounds like you already found the issues?
@danderson happy to support some bare metal and diverse switching (Juniper) + local and global BGP testing at Packet.net. Maybe take advantage of https://github.com/cncf/cluster ?
We've just deployed MetalLB to our new test env. on a 1.9 K8s and eagerly watching #7 and #83. We've got a use case where any and all pods can be publicly routable so we're looking at having an IPv6 only cluster and each pod/svc getting IPv6.
We need non-HTTP svc so planning to use IPs to identify pods when SNI/TLS isn't available (of which we're launching with thousands this summer and expect that to grow to 10s of in 12-18 months.) Aligning some things with K8s 1.10 release for beta IPv6 support and will probably be running alpha versions of stuff in test leading up to launch.
FWIW we use OVH and each server comes with a /64 IPv6 block so when this is in, being able to draw on that pool from each K8s compute node will be ideal. As it stands we have no Go lang expertise but if we can contrib in any other way do let us know. We're comfy with C++/Java/Scala and I'll probably be learning Go this yr since we're committed to K8s.
I just tried this in ARP mode in my k8s-cluster @ home. Works so far, thanks for this great project 👍
I may also deploy this at work in the future for an internal development cluster.
Just jumping in as well. I'm using ARP mode in my home lab with a cluster I set up following Kubernetes the Hard Way to learn Kubernetes. I'm using Weave Net to handle my cluster networking and running on XenServer VMs. I haven't gotten metallb running correctly yet but I'm working on it. :)
We are using MetalLB on a 30 node k8x cluster (internally) in production. So far, no issues.
Just wanted to say thank you for this project! I have spent hours trying to figure out several issues with keepalived-vip before stumbling across MetallLB.. Installed it in 5mins and it just works (and is a more elegant approach, too). Time will tell how stable it is, but no issues whatsoever so far!
Running it on both a 3 OrangePI PC2 cluster and a 3 VM CoreOS cluster with k8s 1.9.
Works like a charm!
Love the ease of installation.
Hi, we will use this when we will launch our ongoing k8s project.
and i am also using this on my cluster at home.
just love it :)
This project is lovely.
I'm using it on my home cluster, and plan to use it on my cluster at work.
We have a old-school deployment at my employer which doesn't afford me the flexibility to setup such niceties as BGP (or really any routing protocol that isn't EIGRP). Most everything that we do is layer 2 separated, so I felt left in the dust by people who got to just specify type: LoadBalancer and off they went.
This project makes k8s fit into my org.
<3
Using this LB implementation in my home lab to learn k8s and look forward to using it in a project at work!
Using this project on my homelab to facilitate Ansible configuration of single container pods is a breeze. Amazing work!
We are using this to try out Kubernetes on our own hardware. Thanks for creating it.
Running it against SRX and MX juniper hardware.
Thanks for making it!
I am evaluating metallb for use in "bare-VM" rather than bare-metal. There is however no difference, the problems are the same.
For testing I "install" metallb by starting the 2 binaries directly on the node vms, i.e not in pods. Due to the simple and elegant architecture this works perfectly fine.
I have learned a lot about Kubernetes by studying metallb. Thanks!
my company started trying out metallb.
we first tried to use it via layer 2 (we use calicoctl and had configured a bgp peer (that was not in use))
however only one client could connect to the service, we had no idea, why. maybe ARP packets were filtered somewhere.
I then removed the bgp peering from calico and used it with metallb which finally worked.
it's really cool to have a metallb.
however it's sad that metallb does not have some kind of LVS way of attaching IPs in layer2 mode, which won't use ARP requests, so it would be useful and less error prone in most networks.
I'm getting my feet wet in Kubernetes at my company. Metallb has proven really useful for repeatable configuration of Ingresses (using nginx-ingress) on available layer2 IPs. Thanks for this useful software!
I'm running it at home, in my testing kubernetes cluster (which is running weave). It's connected to my pfSense router via BGP. This setup would be perfect in a datacenter setup with a specific IP space for DMZ.
If anyone wants the configs, I'd be happy to share.
Thanks for an awesome piece of software!
I am running it in my home-lab and switched to Metallb after using Keepalived-vip for a while and not finding it stable enough. Keepalived-vip would work fine for a while and then lock up, forcing me to manually delete the "master" pod, after which things would start working again. I hope my worries are over now, using Metallb.
We are using MetalLB in our on-premises cluster. At the moment, we run it in ARP mode, but we are working towards BGP mode with our Juniper switches.
MetalLB is really great to have, it alleviated a big part of the envy of Kubernetes on a cloud infrastructure :)
@FireDrunk I'd be curious to know what your configs are - I want to test BGP with my homelab cluster (+ pfsense)
My config (all done via web interface of pfSense):
# This file was created by the package manager. Do not edit!
AS 64512
fib-update yes
holdtime 90
listen on 10.0.24.1
router-id 10.0.24.1
network inet connected
group "kubernetes" {
remote-as 64512
neighbor 10.0.24.40 {
descr "k8s-master-001"
announce all
local-address 10.0.24.1
}
neighbor 10.0.24.41 {
descr "k8s-node-001"
announce all
local-address 10.0.24.1
}
neighbor 10.0.24.42 {
descr "k8s-node-002"
announce all
local-address 10.0.24.1
}
}
deny from any
deny to any
allow from 10.0.24.40
allow to 10.0.24.40
allow from 10.0.24.41
allow to 10.0.24.41
allow from 10.0.24.42
allow to 10.0.24.42
My metallb config:
config:
----
peers:
- peer-address: 10.0.24.1
peer-asn: 64512
my-asn: 64512
address-pools:
- name: default
protocol: bgp
avoid-buggy-ips: true
addresses:
- 10.2.2.0/24
My Kubernetes machines have 10.0.24.40-42, and my BGP subnet is 10.2.2.0, which could very well be something inside the same range.
I've found one flaw in this config, and that is that pfSense always routes to 1 of the hosts (whichever has come online the last I think). It doesn't automatically balance between routes with the same metric.
I've yet to play with BGP multipath.
Currently in the process of deploying it on our infrastructure, 8 cluster around the globe, 600 bare metal nodes, peering against Calico node (patched, #114), Calico in turn peers again TOR switches(Arista/Cisco)
We are using it for datacenter internal IP ranges only as we need a way to limit user access to the external IP pools while keeping the service object(ClusterIP/LoadBalancer with internal IP pool) available to everyone.
We'll either setup some admission controller or use a transparent mode firewall so the IT/infosec people could have the final word on exposing a new service to the world.
I've deployed metallb in layer2 mode to a kubespray 4 node kubernetes cluster, weave networked on terraformed vm's hosted on my home bare metal opennebula kvm IaaS which is running on 4 fanless pc's with 8-16GB and 4-8CPU on each host.
MetalLB is fantastic, I just installed the manifest and followed your LAYER 2 MODE TUTORIAL, then just like magic the LoadBalancer was assigned an external IP. Wow.
I learn and experiment, using all kubernetes features without the cost of expensive cloud based public ip's.
BigUp to all involved.
Installed on my home k8s cluster - worked fine so far (layer2).
We are in the process of switching all our on-premise bare-metal clusters which are not publicly exposed from a commercial load-balancer-platform to Metal-LB - and had no issues with it so far.
We are using BGP via ToR-Routers. Since we have enhanced needs in IPAM, we do not use auto-assignment of addresses but define them statically in the services. Future work would bring a dedicated controller doing our IPAM, which would reduce our use to just the speakers, not the controller anymore.
A big thank you to all authors, maintainers and contributors and especially @danderson, this was exactly what we were looking for atm.
Am using metallb in homelab as well. Working fantastically in layer 2 (arp) mode!
We have our own cloud environment. When we install jenkins x in our Kubernetes cluster, we use metallb in order for the jenkins x to create a LoadBalancer type of service. Thanks for your work.
We are using MetalLB in our k8s cluster in pre-production stage. Thanks for your great job!
installed in my home lab. worked first go!! (layer 2) great job!
Using metallb in pre-production cluster (layer2), and it is working great. We had no problems implementing, and the O&M has been effortless.
We are using metallb in layer-2 mode for on-premise K8 testing.
Hello, we are testing metallb solution on our kubernetes cluster (https://cloud.garr.it/containers/).
Great job! We are very interested in having this solution in production.
@albebeto @alexbarchiesi
I've just finished my PoC proving that I can cluster a legacy app on bare metal k8s using MetalLB!
Installed in my homelab in BGP mode.
(You know this, but joining the happy user list. :))
We are running some experiments with this at section.io
I'm using this in various capacities and it's worked great in each. @FireDrunk one interest I have is putting together some glue between this and pfsense to automatically update the openbgp config based off of nodes coming/going. If you have any input to provide or are interested in the same idea let me know.
Just started using MetalLb for bare metal K8s in test & dev environments. Really positive experience so far. Love the fact that its so simple compared to K8s ingresses - which we found were awkward for bare metal K8s clusters. Watching this project with interest with a view to rolling out to production clusters.
For anyone using pfSense I've created a controller that integrates with metallb to automatically update neighbors: https://github.com/travisghansen/kubernetes-pfsense-controller
I use metallb in myhome lab in L2 mode.
This is so easy to use.
With this, I could make use of my k8s cluster at home.
Can't believe I'm just learning about this project. I use MaaS to provision metal servers to deploy various clusters. I'm super excited to set up MetalLB this weekend with our lab at ClusterGarage.io -- we're just getting started and this will be a great addition.
I've been testing it in my home lab and in the workplace. We're just starting so no real impressions as yet. Will update with more when I've got some more experience with it.
Hello!
I am going to test MetalLB in my home lab environment for studying.
Thank you for your work!
We just setup MetalLB on our k8s cluster.
We use routing to the host w/ BGP unnumbered via Free Range Routing and Cumulus Linux.
We use Calico CNI in policy only mode, and rely on FRR on the host to advertise both the host routes and pod routes into our spine/leaf network.
We updated our MetalLB pod daemonset to use pod networking instead of host networking. We then have MetalLB peer with the host FRR using dynamic BGP peers (and a peer + filter for each host in the MetalLB configmap). This allows MetalLB to announce its IPs into the routing mesh without needing to peer with TOR leaves.
Working great so far! With externaltrafficpolicy: local, we're able to avoid source IP problems by sending traffic only to nodes with service pods/endpoints on them.
Thanks for putting this out in the world, exactly what I was looking for for my test cluster!!!
very nice .. im planning to use this in our onprem cluster
thanks again!
I'm running it at home, in my testing kubernetes cluster (which is running weave). It's connected to my pfSense router via BGP. This setup would be perfect in a datacenter setup with a specific IP space for DMZ.
If anyone wants the configs, I'd be happy to share.Thanks for an awesome piece of software!
i would love to try your configs! im stuck trying to make it work with a cisco router :(
Can't go into details but we are actively using it in an enterprise staged environment and should be taking it into Prod in the coming months. So far all looking good. We have a keepalived based backup plan but to date metallb has been faultless.
I have started using it and doing the configuration with the Layer2
kubectl apply -f https://raw.githubusercontent.com/google/metallb/v0.7.3/manifests/metallb.yaml
kubectl apply -f https://raw.githubusercontent.com/google/metallb/v0.7.3/manifests/example-layer2-config.yaml
our product is multi-platform product and I am responsible for the VMware stack. Still not sure to use the calico or not. Testing it and will open any bugs find.
Using it in production on our load balancer cluster in BGP mode for a month. No issues this far.
I am running a kubespray cluster with MetalLB in layer 2 mode, going to try out BGP with a pfsense soon.
https://github.com/kubernetes-sigs/kubespray/tree/master/contrib/metallb
I set up a 3 node kubernetes cluster on packet.net with MetalLB in BGP mode using elastic public ips. Up next, BGP locally with mikrotik.
I've setup MetalLB in L2-mode on 3 "bare-VMs" using Typhoon, with Calico and CoreOS under the hood.
Despite the issue and solution mentioned at poseidon/typhoon#272, I was unable to get ingress to work as documented.
So instead I installed MetalLB, the Helm chart stable/nginx-ingress and used the (public) IP addresses of the 2 active nodes:
kind: ConfigMap
metadata:
namespace: metallb-system
name: config
data:
config: |
address-pools:
- name: ip-space
protocol: layer2
addresses:
- 192.168.136.142/32
- 192.168.136.143/32The LoadBalancer IP field of the nginx-ingress service:
$ kubectl get svc -n nginx-ingress
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
nginx-ingress-controller LoadBalancer 10.3.16.218 192.168.136.142 80:32672/TCP,443:31315/TCP 3h22m
nginx-ingress-default-backend ClusterIP 10.3.182.96 <none> 80/TCP 3h22mAnyway, thanks for MetalLB! 🏆
Yesterday I got metallb running in a Vagrant 2 node kubernetes cluster using our internal cluster setup tools. Today I'm going to try it out on bare metal where we use Calico and a hardware router with BGP configured. This is so far absolutely the best option for my use cases so I hope this project continues to be supported!
Update: Just wanted to post that I did get MetalLB running on bare metal and it was soooooo much easier than setting up ingress, and it suits all my needs. Yay! Thanks for the awesome software!
Using it with kube-router on k8s 1.13 with BGP mode to an Ubiqutity EdgeRouter 4. Its great, its fast. I had some problems with older versions of kube-router, but they seem to work nice together with latest versions.
Just installed on my kubesprayed 5 node cluster w/calico. Layer 2 mode worked with almost no effort. Moving a lot of web sites from unorchestrated containers to k8s, and this looks like a great addition to the new architecture. Project looks very well done!
hi, I am using metalLB and it is running well.
I want to understand the metallb how to work, and I should how to learn this source code, Excause me, is there any architecture diagram help me study? Thanks!
Just gave it a try, working like a charm. Don't want to involve a IaaS platform for now.
I am using MetalLB !! love it
Finally working loadbalancers for on-premise setups. Thank you so much.
Using it on our test and production cluster, ARP mode. Stable as a rock !
Thx for the hard work !
I'm using it from start in our K8S proof-of-concept which is almost finished, and it looks like MetalLB BGP will survive the scrutiny and move into production. Currently working hard on attempting to use it combined with ingress-nginx (working fine) to do some basic traffic filtering (not yet working).
Good job and thanks for the hard work!
Using MetalLB in our reference "k8s on bare metal at Packet" demo and also recently merged into Kubespray on bare metal support:
https://github.com/kubernetes-sigs/kubespray/blob/master/docs/packet.md
https://github.com/packet-labs/kubernetes-bgp
Thanks for a great project!
We're an EU research institute and currently setting up for our transfer service.
Will be using metalb in BGP mode to replace our own BGP service router currently in prod. The ability to peer with multiple sets of TOR switches is clutch ! Thanks for the awesome work on this project
Thank you guys for this well-made project!
Using it since a few days in ARP mode for my bare metal homelab, and will eventually deploy it on a test cluster at work.
I really enjoyed the installation via helm, having very intuitive configuration options, and an easy startup.
Using MetalLB in my home lab whilst trying to learn how to deploy a production grade cluster at work.
Just figured out how to get my containerised local DNS running with UDP and TCP exposed on the same IP.
Cheers.
I'll be experimenting with metalLB. Hoping to deploy it into our production cluster
We're using metallb in our ci/cd k8s on bare metal cluster. Thanks a lot
MetalLB is an absolute life saver and makes testing and running K8S clusters not hosted in one of the main cloud providers sane. We're using it for deployments at NASAJPL and DARPA.
What an awesome solution! Kudos and many thanks for developing MetalLB - just started it using it and impressed at how easy it was to setup on Vagrant with Rancher 2.0 :) Although Rancher doesn't see the load balancer, so can't create a service - I'll keep digging in on the issue.
Cheers, David
Just a tiny homelabber who wants to avoid PENDING in k8s. Thanks.
I am using it quite a fair bit in my local development cluster along with Rancher, Traefik and Linkerd.
Thanks a ton for the awesome solution, no more pesky <pending> LoadBalancers.
I am using it in my test cluster. Awesome! Waiting for your GA version.
It would be better if I could specify external IP（In the IP poll define by configmap） in the Service Yaml file.
@liaolinrong you already can define your loadBalancerIP in the service, ip assignment by metallb-controller is only done if no manual address is specified. Validation against pool ip range is still done.
@FireDrunk would you mind sharing your MetalLB+pfSense configuration?
@teralype and @FireDrunk if you're using pfSense you may find this of interest: https://github.com/travisghansen/kubernetes-pfsense-controller
Among other things it automatically configures openbgp (could add support for other servers if needed) neighbors for you.
We're using in on our lab env, which will be hopefully rebuild soon as production.
- rke 0.22 based kubernetes 1.14 on metal (+rancher2 on top)
- calico 3.7.0 in L3 mode
- metallb in L3 behind calico peering ebgp on 127.0.0.1 with some inspiration from #114 (comment)
works like a charm ;-)
I'm running a small cluster for an educational institution: Kubernetes 1.14.1, deployed via Rancher's inbuilt RKE) on a mix of bare-metal and VMs (4 nodes, ≈ 130 CPU cores, 8 GPUs, ≈ 460 GiB RAM), Canal, Metallb behind pfsense (BGP configured without the controller).
If possible, I'd very much prefer to run L2 mode to reduce complexity. But that would require an option for Metallb to populate the LB's external IP with one from the public subnet I mapped to the local IP range Metallb hands out.
Layer 2 mode
- RKE v0.2.2 Kubernetes 1.14 on BHyve VMs
- K3S v0.5.0 Kubernetes on ARM (S905X AArch64)
I started using metalLB 0.7.x with Rancher 2.2.2, not working. After Rancher upgrade to 2.2.3, it works like a charm !
Really was waoou when getting an external IP.
- 5 clusters
- About 21 VMware VMs.
- Full on prem
- Layer 2
Needed a load balancer for my cluster at home. Found your project, spent less then 5 minutes to set it up ... works like a charm. Thank you for your awesome work.
Using it for 8 clusters running in vmware. Layer 2 mode.
Using in BGP mode in openshift test env
Thanks for great software
Using in Layer 2 mode in k8s test env
Thanks !
Using it in Layer 2 mode on my home i3 single-node cluster.
Awesome work! Thanks.
I use it at both work and my home-cluster. Great stuff!
We are using it in our development environments as a replacement for the stock LB which ships with the Kube instance installed with Docker for Windows Desktop.
The standard LB infrastructure does not allow TCP and UDP comms on the same port and it is also quite clunky in the way it binds services to localhost.
Metal LB allows us to bind multiple services on different ports and expose them on all network interfaces including DockerNAT, which enables uniform microservice visibility (same IP) from within the cluster as well as from within the dev box.
Bottom line is, without Metal LB, we would not be able to develop/test/debug our services on our Windows hosts and have a bidirectional discoverability between the service being developed on the Windows box and its dependency services running in the local Kube cluster.
Just found it and started using on my small production cluster (2 nodes) and larger research cluster (11 nodes). Switched from what I was using before because you support the ability to request IPs and do something useful with the externalTrafficPolicy parameter. I use in Layer2 mode for now ... but may investigate more advanced modes soon.
Thanks for the great tool -- worked first time, no issues!
I found MetalLB when coming to the surprise that k8s doesn't provide external networking/LB when not having a cloud provided cluster.
Working with it on an on prem k8s lab cluster and it works easily and well (once you put the config in the right namespace :) whoops). Thanks to @danderson for helping finding my silly mistake.
Much appreciated for creating a suitable option for on prem clusters.
We have an in-house cluster of servers and we're using MetalLB to deploy individual Dev. stack on it so that all the development and tests are running on as close a system as our production kubernetes cluster.
Truly appreciate this project! Thank you
More DevOps and SRE teams need to know about MetalLB.
Many new to K8s are left with the sense that the layer2 external ip pool allocation is only if you're using kubernetes in a public cloud or with expensive enterprise solutions. An in-kubernetes-native solution to allocate IP addresses from address pools that are decoupled from the kubernetes NodePort/HostNetworking layer has been a missing piece.
For a lot of shops needing to integrate their emerging K8s stacks with traditional infrastructure provisioning processes ( another team issues DNS A records and SSL certificates), this has been a game changer.
For me, I was able to give ( or have metallb allocate and plumb, rather) an ingress controller's LoadBalancer service a single durable IP "External IP" address ( with a lifecycle decoupled from the lifecycle of the Cluster or any individual Node's own interface addresses ). Before this, whenever any of the Node's IP or NodePort for a given Services listening sockets changed, which happens a lot, I would have to have our DNS folks update DNS; and then I would have to also maintain NodePort-based backend configurations on a separate out-of-k8s linux box running nginx/apache
Now, with MetalLB, the full datacenter model is all built in at the tiniest footprint level across all flavors - by inherently being native - distributions of k8s.
The possibilities are endless now.