Metrics are unavailable

[Michael Bishop]

For both pre-existing pods, etc and the self-installed ElasticKube services, neither metrics nor events seem to be available.

The elastickube-api logs show the following:

INFO:root:Initializing KubeWatcher
DEBUG:root:check_permissions for user [e-mail] and operation watch on kube watch
INFO:root:Starting watch KubeWatcher for action instance with params {'kind': u'Pod', 'namespace': u'default', 'name': u'redis-9yp2t'}
DEBUG:root:Starting watch instance connected
DEBUG:root:Added watcher for resource Pod and params {u'resourceVersionEventList': u'8150', 'kind': u'Pod', u'resourceVersionPod': u'120095', 'resourceVersionLogList': 1459676202.185422, 'uid': u'fc3f3aba-f761-11e5-b53d-42010af000d6', 'namespace': u'default', 'resourceVersionMetricList': 1459676202.162161, 'name': u'redis-9yp2t'}
DEBUG:root:Added watcher for resource EventList and params {u'resourceVersionEventList': u'8150', 'kind': u'Pod', u'resourceVersionPod': u'120095', 'resourceVersionLogList': 1459676202.185422, 'uid': u'fc3f3aba-f761-11e5-b53d-42010af000d6', 'namespace': u'default', 'resourceVersionMetricList': 1459676202.162161, 'name': u'redis-9yp2t'}
WARNING:root:Disconnected from kubeclient.
**ERROR:root:HTTP 410: Gone**
None
DEBUG:root:InstancesWatcher data_callback
**WARNING:root:Unexpected message from Kubernetes: {u'status': u'Failure', u'kind': u'Status', u'code': 410, u'apiVersion': u'v1', u'reason': u'Gone', u'message': u'too old resource version: 120095 (127572)', u'metadata': {}}**

If I can provide any additional info, more than happy to.

Loving the UI, by the way. This should be the default on K8S!

[Arnaud]

I am looking now to see what is happening. The errors you are seeing are related to events which are not available anymore in Kubernetes for "old" resources. You can verify this by running:

• kubectl describe pods --namespace=kube-system
  You can check that Kubernetes is not returning any event.

I will keep you posted for the metrics issue.

[Michael Bishop]

Arnaud, Makes perfect sense (on the events issue). And come to think of it, about the only time I pay attention to the events is to troubleshoot an initial launch so under the principle of "no news is good news", a lack of recent events is a good thing. :)

Thanks for the quick response -- really loving the UI and find myself using it as my primary dashboard already. When I grab a few hours, I'll dig in on some of these open items if they're still open and see if I can pitch in.

Cheers,
Michael

[Arnaud]

@mdbishop I have made some research and found that GKE is using the v1.0.0 release of Heapster, which seems to have deprecated some of the API endpoints we are using, especially the one referred by this issue: kubernetes-retired/heapster#1107

I have reproduced internally and will work on addressing these API changes.

[Michael Bishop]

Kubedash itself seems to deal with this issue by deploying an old version of heapster alongside the new for kubedash's use when running on a > 1.1 cluster.

--- > see deploy/bundle.yaml in the repo (and the repo's readme which also mentions this)

Hope this helps.

[Arnaud]

This is has been fixed and tested against the latest release of Kubernetes and heapster version from https://github.com/kubernetes/kubernetes/tree/v1.2.4/cluster/addons/cluster-monitoring.