failed to start elastickube
onyxet opened this issue · comments
Hi! When i try start elastickube I have some errors in logs like this:
2016-03-14T14:10:13.996066529Z ERROR:root:Unexpected error executing GitSync sync loop.
2016-03-14T14:10:13.996096505Z Traceback (most recent call last):
2016-03-14T14:10:13.996102849Z File "/opt/elastickube/charts/sync/init.py", line 40, in initialize
2016-03-14T14:10:13.996108764Z yield GitSync(motor_client.elastickube).sync_loop()
2016-03-14T14:10:13.996114195Z File "/opt/elastickube/charts/sync/repo.py", line 48, in init
2016-03-14T14:10:13.996119751Z self.repo = Repo.clone_from(self.url, REPO_DIRECTORY)
2016-03-14T14:10:13.996125135Z File "/usr/local/lib/python2.7/site-packages/git/repo/base.py", line 885, in clone_from
2016-03-14T14:10:13.996130584Z return cls._clone(git, url, to_path, GitCmdObjectDB, progress, **kwargs)
2016-03-14T14:10:13.996135803Z File "/usr/local/lib/python2.7/site-packages/git/repo/base.py", line 831, in _clone
2016-03-14T14:10:13.996141116Z finalize_process(proc)
2016-03-14T14:10:13.996146093Z File "/usr/local/lib/python2.7/site-packages/git/util.py", line 155, in finalize_process
2016-03-14T14:10:13.996160369Z proc.wait()
2016-03-14T14:10:13.996166154Z File "/usr/local/lib/python2.7/site-packages/git/cmd.py", line 319, in wait
2016-03-14T14:10:13.996171615Z raise GitCommandError(self.args, status, self.proc.stderr.read())
2016-03-14T14:10:13.996177006Z ValueError: I/O operation on closed file
My conf of elastickube-server.yaml is:
apiVersion: v1
kind: ReplicationController
metadata:
name: elastickube-server
namespace: kube-system
labels:
name: elastickube-server
spec:
replicas: 1
selector:
name: elastickube-server
template:
metadata:
labels:
name: elastickube-server
spec:
containers:
- name: elastickube-api
image: registry.it.loc/elastickube-api:1.0
resources:
limits:
cpu: 100m
memory: 128Mi
volumeMounts:
- name: elastickube-code
mountPath: /opt/elastickube
- name: elastickube-run
mountPath: /var/run
- name: elasticbox-home-user
mountPath: /home/elasticbox
- name: elastickube-charts
image: registry.it.loc/elastickube-charts:1.0
resources:
limits:
cpu: 50m
memory: 64Mi
volumeMounts:
- name: elastickube-code
mountPath: /opt/elastickube
- name: elastickube-charts
mountPath: /var/elastickube/charts
- name: elastickube-nginx
image: registry.it.loc/elastickube-nginx:1.0
resources:
limits:
cpu: 10m
memory: 32Mi
volumeMounts:
- name: elastickube-code
mountPath: /opt/elastickube
- name: elastickube-run
mountPath: /var/run
ports:
- containerPort: 80
hostPort: 8084
name: http
protocol: TCP
nodeSelector:
kubernetes.io/hostname: aws-hyperkube
volumes:
- name: elastickube-code
hostPath:
path: /home/sybase/elasticbox/src/
- name: elastickube-charts
hostPath:
path: /home/sybase/elasticbox/src/charts
- name: elastickube-run
hostPath:
path: /var/run/elastickube
- name: elasticbox-home-user
hostPath:
path: /home/sybase/elasticbox
as I understand it , this code tries to load app files, but I have them when creating pod via mounting. Where is my mistake?
Hi,
I have seen this error when there is a git repo already in the folder where the charts container is trying to clone. Can you log-in into the container and check the contents of /var/elastickube/charts? Is anything in there?
Another reason might be that the charts service cannot write into the local folder. This first release of elastickube rc is pre-configured to use a local folder on the node where the pod is running. I would alos recommend that if your cluster has support for persistent volumes use a persistent volume instead of a hostPath.
In /var/elastickube/charts I have this files:
init.py init.pyc entrypoint.sh server.py sync
Have elastickube solution for deployment in closed clusters? Where I don`t need clone repo from internet. I have own registry and kubernetes cluster in private network. What the best way for deploy elastickube? I tryed run it on test cluster with this
curl -s https://elastickube.com | bash
but had LB error. Where logs from this?
UPDATE: Find logs in home dir but didn`t saw why I had LB error
The recommended way to deploy ElasticKube is through the url. If you want to customize the deployment you can download the script using curl -s elastickube.com -o install.sh and modify the resource definitions as desired. (I will push the script today in the repo so anybody can modify)
Regarding the load balancer. We assume your kubernetes is capable of deploying a loadbalancer service. If this is not the case you can still check out ElasticKube through the proxy service. Deploy this service:
apiVersion: v1
kind: Service
metadata:
name: elastickube-server
namespace: kube-system
labels:
name: elastickube-server
spec:
ports:
- port: 80
targetPort: 80
selector:
name: elastickube-server
And access it through the Kuberentes api service:
http://<kubernetes_api>/api/v1/proxy/namespaces/kube-system/services/elastickube-server
Finally I think the issue you were experiencing with the charts service is related to not correctly mounting the drives. The yaml files you are using are designed to support a development environment where we mount the git repo code and reload the processes inside the containers every time the code changes. If you want to contribute to the code. I can help you setup the development environment.
Thanks! Solved issue with ValueError: I/O operation on closed file but now I have this in logs :
AutoReconnect: [Errno 111] Connection refused
ERROR:root:Unexpected error executing GitSync sync loop.
Traceback (most recent call last):
File "/opt/elastickube/charts/sync/init.py", line 40, in initialize
yield GitSync(motor_client.elastickube).sync_loop()
File "/usr/local/lib/python2.7/site-packages/tornado/gen.py", line 1008, in run
value = future.result()
File "/usr/local/lib/python2.7/site-packages/tornado/concurrent.py", line 232, in result
raise_exc_info(self._exc_info)
File "/usr/local/lib/python2.7/site-packages/tornado/gen.py", line 1014, in run
yielded = self.gen.throw(_exc_info)
File "/opt/elastickube/charts/sync/repo.py", line 63, in sync_loop
settings = yield Query(self.database, "Settings").find_one() or dict()
File "/usr/local/lib/python2.7/site-packages/tornado/gen.py", line 1008, in run
value = future.result()
File "/usr/local/lib/python2.7/site-packages/tornado/concurrent.py", line 232, in result
raise_exc_info(self._exc_info)
File "/usr/local/lib/python2.7/site-packages/tornado/gen.py", line 1014, in run
yielded = self.gen.throw(_exc_info)
File "/opt/elastickube/data/query.py", line 49, in find_one
document = yield self.database[self.collection].find_one(self._generate_query(criteria), projection)
File "/usr/local/lib/python2.7/site-packages/tornado/gen.py", line 1008, in run
value = future.result()
File "/usr/local/lib/python2.7/site-packages/tornado/concurrent.py", line 232, in result
raise_exc_info(self._exc_info)
File "", line 3, in raise_exc_info
in dir /var/elastickube/charts:
Makefile alpine elasticsearch example-todo minecraft nginx-alpine proxy-2-service redis-standalone selenium-hub ubuntu-debootstrap
README.md cassandra elasticsearch-1 jenkins mysql owncloud rabbitmq redmine spark
_test docker-registry example-nginx memcached nginx postgresql redis-cluster riak storm
It`s for helm?
where i can see git command for cloning repo in charts?
rest of containers working properly without errors in logs.
I believe the problem is that your container does not have internet access and is failing to clone the repo.
We use GitPython to interact with the git repo. You can see the code here:
https://github.com/ElasticBox/elastickube/blob/master/src/charts/sync/repo.py
Thanks for feedback. I`ll try reproduce this on another cluster with public access and let you know
That would be great. If we pin point the issue I will add better error messages to the install script.
curl -s elastickube.com -o install.sh returns Moved Permanently.
curl -L -s elastickube.com -o install.sh returns HTML output...
@albertoarias I tried run elastickube on node with full access to internet and had same issue with "connection refused". In container nginx I saw next logs:
2016-03-23T16:03:33.233899220Z 2016/03/23 16:03:33 [error] 6#6: *5 connect() to unix:/var/run/elastickube-api.sock failed (111: Connection refused) while connecting to upstream, client: 10.1.205.245, server: , request: "GET /api/v1/auth/providers HTTP/1.1", upstream: "http://unix:/var/run/elastickube-api.sock:/api/v1/auth/providers", host: "10.59.101.34", referrer: "http://10.59.101.34/"
Have you any ideas? thanks for feedback
And when I try start api in container with
/bin/bash /opt/elastickube/api/entrypoint.sh
I have this
Initializing
.........................
I think trouble with API. Where there may be a problem?
@warden I just double checked and the output should be an script. Are you using a proxy server by any chance? We use the user-agent to detect the curl client so maybe that is the issue. You can access the install script directly from here:
https://raw.githubusercontent.com/ElasticBox/elastickube/master/build/deploy.sh
@onyxet The initializing message happens when trying to connect to mongo for the first time. If your container cannot connect is because the following reasons:
a) Mongo is not running
b) The service is not created
c) The pod was created before the mongo service was created
d) The mongo service was recreated after the elastickube-server was created
Can you try logging into the container and see if you can ping the mongo service:
kubectl --namespace=kube-system exec elastickube-server- -c elastickube-api -ti -- bash
You can check if the environment variable with the mongo address is present using the 'env' command.
@onyxet Can you also send me gist with the replication controller yaml that you used?