[BUG] LeaderElection is misconfigured in routecontroller
tcdowney opened this issue · comments
Tim Downey commented
Summary
As part of upgrading Golang and other dependencies in routecontroller we uncovered a bug in how we were configuring LeaderElection. It was enabled on the routecontroller
in #175193243, but had several issues.
The first issue was due to kubernetes-sigs/controller-runtime#445 requiring that LeaderElectionID
and LeaderElectionNamespace
be explicitly set.
We fixed this first issue in a8d9323, but leader election is still failing since we don't have the necessary RBAC since it now is trying to use the leases.coordination.k8s.io
resource.
E0319 16:14:23.174545 1 leaderelection.go:325] error retrieving resource lock cf-system/cf-k8s-networking-routecontroller: leases.coordination.k8s.io "cf-k8s-networking-routecontroller" is forbidden: User "system:serviceaccount:cf-system:routecontroller" cannot get resource "leases" in API group "coordination.k8s.io" in the namespace "cf-system"
Tim Downey commented
We manually validated this by...
- Scaled the
routecontroller
to 2 replicas and observed the logs for eachPod
. - We observed that one was doing work and reconciling
Route
resources. The other was not doing work because it did not have the lease. It logged something like:I0322 17:25:50.999788 1 leaderelection.go:243] attempting to acquire leader lease cf-system/cf-k8s-networking-routecontroller...
- We confirmed that integration tests were no passing for
routecontroller
.