Problem starting ElasticKube Server
bergtwvd opened this issue · comments
I am running an "unsecure" Kubernetes install (one master, three nodes), i.e. not certificates.
I do curl -s https://elastickube.com | bash -s -- -u http://10.10.10.150:8080 from an Ubuntu machine on which I only have kubectl installed. *.150 is the address of the master.
Script runs with following output:
_____ _ _ _ _ __ _
| ____| | __ _ ___| |_(_) ___| |/ / _| |__ ___
| _| | |/ _` / __| __| |/ __| ' / | | | '_ \ / _ \
| |___| | (_| \__ \ |_| | (__| . \ |_| | |_) | __/
|_____|_|\__,_|___/\__|_|\___|_|\_\__,_|_.__/ \___| by ElasticBox
Checking kubectl is available [ ✓ ]
Verifying Kubernetes cluster [ ✓ ]
Setting up elastickube-server svc [ ✓ ]
Setting up elastickube-mongo svc [ ✓ ]
Setting up elastickube-mongo [ ✓ ]
Setting up elastickube-server [ ✓ ]
WARNING: LoadBalancer Ingress not detected, please ensure the address is accessible from outside the cluster. Check http://kubernetes.io/docs/user-guide/ingress/ for more information.
Waiting for LB to be ready [ ✓ ]
ElasticKube has been deployed!
Please complete the installation here: http://10.254.80.252
However , the ElasticKube Server won't start.
When I check the Kube dashboard at the elastickube server I see in the log:
2016-12-04T16:35:08.247532000Z Traceback (most recent call last):
2016-12-04T16:35:08.346998000Z File "/opt/elastickube/diagnostics/diagnostics.py", line 415, in <module>
2016-12-04T16:35:08.347292000Z run_server()
2016-12-04T16:35:08.347510000Z File "/opt/elastickube/diagnostics/diagnostics.py", line 389, in run_server
2016-12-04T16:35:08.347711000Z settings_from_env(settings, os.environ)
2016-12-04T16:35:08.348059000Z File "/opt/elastickube/diagnostics/diagnostics.py", line 319, in settings_from_env
2016-12-04T16:35:08.348391000Z with open(token_path, 'r') as f:
2016-12-04T16:35:08.348602000Z IOError: [Errno 2] No such file or directory: '/var/run/secrets/kubernetes.io/serviceaccount/token'
What next?
Hello,
ElasticKube manages the cluster from the inside using a Secret called "token" available by default in the kube-system namespace. This token is required to do most of the operations.
We need to be sure that the "token" Secret is present and functional.
It seems that the secret token is not available for ElasticKube:
http://kubernetes.io/docs/user-guide/accessing-the-cluster/#accessing-the-api-from-a-pod
Can you please check if the secret token is available using these commands?
kubectl get secrets
kubectl describe secrets
Regards.
[bergtwvd@master ~]$ kubectl get serviceAccounts
NAME SECRETS AGE
default 0 7d
[bergtwvd@master serviceAccount]$ kubectl describe secrets
Name: registrypullsecret
Namespace: default
Labels: <none>
Annotations: <none>
Type: kubernetes.io/dockerconfigjson
Data
====
.dockerconfigjson: 96 bytes
I created this token for my private Docker Registry.
There are no other tokens.
To get Elastic going, what are the instructions I should pass into Kubernetes?
The default serviceAccounts for the default namespace does not allow to run ElasticKube there.
In which namespace are your ElasticKube pods, rcs and services? By default they are created in the kube-system namespace which has a serviceAccount that has enough privileges. Do you have the kube-system namespace?
If you don't have it or the pods are not there.
Please, run the following commands:
kubectl get serviceaccounts --namespace=kube-system
kubectl get secrets --namespace=kube-system
kubectl get pods --namespace=kube-system
kubectl get services --namespace=kube-system
kubectl get secrets --namespace=kube-system {{ NAME OF THE SECRET USED BY ELASTICKUBE IF ANY }}Did you do any customization of the installation sh script?
All this should be already in the cluster and ElasticKube just uses the default infrastructure. It would be really useful to know the way you installed your cluster to improve our documentation, because the shell script should have taken care of all this for you.
I just re-ran the command directly on my master kube:
curl -s https://elastickube.com | bash -s -- -u http://10.10.10.150:8080
I have not done any adaptations to the deploy script.
It ends with the message:
ElasticKube has been deployed! Please complete the installation here: http://10.254.186.132
Output from the various commands:
[bergtwvd@master elastickube]$ kubectl get serviceaccounts --namespace=kube-system
NAME SECRETS AGE
default 0 7d
[bergtwvd@master elastickube]$ kubectl get secrets --namespace=kube-system
[bergtwvd@master elastickube]$ kubectl get pods --namespace=kube-system
NAME READY STATUS RESTARTS AGE
elastickube-mongo-mb7aa 1/1 Running 0 4m
elastickube-server-qeb5s 3/4 CrashLoopBackOff 2 4m
heapster-2676267885-g5v9y 1/1 Running 1 1d
kube-dns-v11-ugj5r 4/4 Running 1 1d
kubernetes-dashboard-1297179873-ooig3 1/1 Running 1 1d
monitoring-grafana-4128421004-j8gks 1/1 Running 1 1d
monitoring-influxdb-2486686895-y035z 1/1 Running 1 1d
[bergtwvd@master elastickube]$ kubectl get services --namespace=kube-system
NAME CLUSTER-IP EXTERNAL-IP PORT(S) AGE
elastickube-mongo 10.254.138.219 <none> 27017/TCP 5m
elastickube-server 10.254.186.132 80/TCP 5m
heapster 10.254.157.198 <none> 80/TCP 6d
kube-dns 10.254.254.254 <none> 53/UDP,53/TCP 7d
kubernetes-dashboard 10.254.63.125 nodes 80/TCP 7d
monitoring-grafana 10.254.82.162 nodes 80/TCP 6d
monitoring-influxdb 10.254.68.184 <none> 8086/TCP 6d
For installing Kubernetes I followed the instructions at
http://kubernetes.io/docs/getting-started-guides/centos/centos_manual_config and some from http://severalnines.com/blog/installing-kubernetes-cluster-minions-centos7-manage-pods-services.
To get the install to work I had to remove ServiceAccount from KUBE_ADMISSION_CONTROL in /etc/kubernetes/apiserver (see 2nd link above). This workaround has been suggested in quite a few posts (and at kubernetes/kubernetes#11222). I am using plain HTTP, no certificates. Looking at this issue, this seems to be the problem, but I do not know how to fix this in my current install.