Giters

k8snetworkplumbingwg / whereabouts

A CNI IPAM plugin that assigns IP addresses cluster-wide

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

ip-reconciler panic: runtime error: slice bounds out of range [:3] with length 2

infinitydon opened this issue · comments

commented

Hi,

We encountered an issue with the ip-reconciler but unfortunately the logs from the ip-reconciler pod is not pointing to what the error could be:

2022-01-17T15:14:31Z [debug] NewReconcileLooper - Kubernetes config file located at: /host/etc/cni/net.d/whereabouts.d/whereabouts.kubeconfig
2022-01-17T15:14:31Z [debug] successfully read the kubernetes configuration file located at: /host/etc/cni/net.d/whereabouts.d/whereabouts.kubeconfig
2022-01-17T15:14:33Z [debug] listing IP pools
panic: runtime error: slice bounds out of range [:3] with length 2

goroutine 1 [running]:
github.com/dougbtv/whereabouts/pkg/reconciler.getFlatIPSet(0x0, 0x0, 0x0, 0x0, 0xc0057eec60, 0x2f, 0xc0057eec90, 0x2a, 0xc005815529, 0x5, ...)
        /go/src/github.com/dougbtv/whereabouts/pkg/reconciler/wrappedPod.go:74 +0x50b
github.com/dougbtv/whereabouts/pkg/reconciler.wrapPod(...)
        /go/src/github.com/dougbtv/whereabouts/pkg/reconciler/wrappedPod.go:28
github.com/dougbtv/whereabouts/pkg/reconciler.indexPods(0xc006b00000, 0x5b8, 0x666, 0xc00282c9c0, 0x8)
        /go/src/github.com/dougbtv/whereabouts/pkg/reconciler/wrappedPod.go:50 +0x2c6
github.com/dougbtv/whereabouts/pkg/reconciler.NewReconcileLooper(0x7ffde3094947, 0x38, 0x191ed20, 0xc0002c9260, 0xc0002c9260, 0xc00053e640, 0x2d)
        /go/src/github.com/dougbtv/whereabouts/pkg/reconciler/iploop.go:53 +0x36b
main.main()
        /go/src/github.com/dougbtv/whereabouts/cmd/reconciler/ip.go:24 +0x196

What we think triggered was a wrong next-hop gateway (it was not within the subnet range of the ip pool) in one of the net-attach-def, this was corrected but the ip-reconciler keeps going into crashloopback/error after the configuration was corrected.

We have checked all the ip-pools and they are in order:

[ec2-user@ip-10-0-0-53 whereabouts]$ kubectl get ippools -A
NAMESPACE     NAME             AGE
kube-system   10.220.10.0-24   40d
kube-system   10.220.11.0-24   40d
kube-system   10.220.13.0-24   38d
kube-system   10.220.14.0-24   31d
kube-system   10.220.15.0-24   39d
kube-system   10.220.16.0-23   39d
kube-system   10.220.20.0-23   39d
kube-system   10.220.24.0-22   39d
commented

Are you using image ip-reconciler:latest? If true, revert to the stable tree to fix this.

commented

Thanks @ttres - We found out that some worker-nodes were running old container image versions because the latest flag was been used, we have set the image version specifically to use actual stable release version, this will also help to track issues with respect to the whereabout version..