panic in findLBEndpointSandbox()

[Xinfeng Liu]

We got a panic in docker engine, stack strace:

Oct 15 21:24:18 docker03 dockerd: panic: runtime error: invalid memory address or nil pointer dereference
Oct 15 21:24:18 docker03 dockerd: [signal SIGSEGV: segmentation violation code=0x1 addr=0x154 pc=0x558695958ae7]
Oct 15 21:24:18 docker03 dockerd: goroutine 1533 [running]:
Oct 15 21:24:18 docker03 dockerd: sync.(*Mutex).Lock(...)
Oct 15 21:24:18 docker03 dockerd: /.GOROOT/src/sync/mutex.go:74
Oct 15 21:24:18 docker03 dockerd: github.com/docker/docker/vendor/github.com/docker/libnetwork.(*endpoint).Name(0x0, 0x0, 0x0)
Oct 15 21:24:18 docker03 dockerd: /root/rpmbuild/BUILD/src/engine/.gopath/src/github.com/docker/docker/vendor/github.com/docker/libnetwork/endpoint.go:286 +0x37
Oct 15 21:24:18 docker03 dockerd: github.com/docker/docker/vendor/github.com/docker/libnetwork.(*network).findLBEndpointSandbox(0xc028c56c40, 0xc0305c5e00, 0xc029213780, 0xc014745038, 0x558694b41101)
Oct 15 21:24:18 docker03 dockerd: /root/rpmbuild/BUILD/src/engine/.gopath/src/github.com/docker/docker/vendor/github.com/docker/libnetwork/service_linux.go:72 +0x159
Oct 15 21:24:18 docker03 dockerd: github.com/docker/docker/vendor/github.com/docker/libnetwork.(*network).rmLBBackend(0xc028c56c40, 0xc00f563b80, 0x10, 0x10, 0xc0172a6000, 0x100)
Oct 15 21:24:18 docker03 dockerd: /root/rpmbuild/BUILD/src/engine/.gopath/src/github.com/docker/docker/vendor/github.com/docker/libnetwork/service_linux.go:181 +0x58
Oct 15 21:24:18 docker03 dockerd: github.com/docker/docker/vendor/github.com/docker/libnetwork.(*controller).rmServiceBinding(0xc0017ac500, 0xc010cfbbf0, 0x21, 0xc019be1cc0, 0x19, 0xc019be1ca0, 0x19, 0xc029213580, 0x40, 0xc00047f380, ...)
Oct 15 21:24:18 docker03 dockerd: /root/rpmbuild/BUILD/src/engine/.gopath/src/github.com/docker/docker/vendor/github.com/docker/libnetwork/service_common.go:379 +0xc44
Oct 15 21:24:18 docker03 dockerd: github.com/docker/docker/vendor/github.com/docker/libnetwork.(*controller).handleEpTableEvent(0xc0017ac500, 0x5586975579a0, 0xc0247248c0)
Oct 15 21:24:18 docker03 dockerd: /root/rpmbuild/BUILD/src/engine/.gopath/src/github.com/docker/docker/vendor/github.com/docker/libnetwork/agent.go:964 +0x12d2
Oct 15 21:24:18 docker03 dockerd: github.com/docker/docker/vendor/github.com/docker/libnetwork.(*controller).handleTableEvents(0xc0017ac500, 0xc00d9a1b60, 0xc004214430)
Oct 15 21:24:18 docker03 dockerd: /root/rpmbuild/BUILD/src/engine/.gopath/src/github.com/docker/docker/vendor/github.com/docker/libnetwork/agent.go:831 +0x53

The reason is obvious:
https://github.com/moby/libnetwork/blob/master/service_linux.go#L72

	if ep == nil {
		return nil, nil, fmt.Errorf("Load balancing endpoint %s(%s) removed from %s", ep.Name(), ep.ID(), n.ID())
	}

Docker version (19.03.11)

[Xinfeng Liu]

Closing. Found there's a fix #2585 which just got merged

Side note: this issue happens when a swarmkit service keeps failing to start.