git push deis master , timeout issue on GKE 1.8.8-gke.0 version
sachinkarale opened this issue · comments
I am getting below issue on GKE 1.8.8-gke.0 version , not able to understand what's going wrong.
"git push deis master
ssh: connect to host deis-builder.35.192.188.163.nip.io port 2222: Connection timed out
fatal: Could not read from remote repository.
Please make sure you have the correct access rights
and the repository exists. "
Connection timed out indicates that the builder is not reachable through the IP you're trying to reach it at. Assuming that is the correct IP and you're able to reach controller there, and authenticate with the API...
Are you using the traditional router or experimental native ingress? If using ingress, currently I believe the correct way to set up ssh port for builder, is usually to expose it with a NodePort or ClusterIP/LoadBalancer service (as in, by hand... as in probably undocumented).
If using router, then it should just work though... that is the main advantage of router at this point AFAIK is that it does not require additional steps to reach the builder.
There's a list of things we can check with you; is builder healthy, does the log indicate it has been stable or is it restarting, what does kubectl --namespace=deis get pods and get services look like?
Thanks for submitting your issue to the forked repos!
Thanks @yebyen , I solved this issue . I was trying from google cloud shell as there is limitation of ports for outbound traffic . I did gcloud setup on my local machine , now it's working fine but facing another issue for git push deis master.
The issue is :
remote: 2018/04/23 11:09:07 Error running git receive hook [Build pod exited with code 1, stopping build.]
To ssh://git@deis-builder.35.184.198.131.nip.io:2222/timely-playroom.git
! [remote rejected] master -> master (pre-receive hook declined)
error: failed to push some refs to 'ssh://git@deis-builder.35.184.198.131.nip.io:2222/timely-playroom.git'
I did some google search on it , as well as I checked deis controller logs , I mentioned those logs as below.
INFO Pushing Docker image 127.0.0.1:5555/example-dockerfile-http:v2
INFO [example-dockerfile-http]: received unexpected HTTP status: 500 Internal Server Error
10.52.2.13 "POST /v2/apps/example-dockerfile-http/builds/ HTTP/1.1" 400 71 "Deis Client v2.18.0"
ERROR:root:received unexpected HTTP status: 500 Internal Server Error
Traceback (most recent call last):
File "/app/api/models/release.py", line 89, in new
release.publish()
File "/app/api/models/release.py", line 136, in publish
publish_release(source_image, self.image, deis_registry, self.get_registry_auth())
File "/app/registry/dockerclient.py", line 195, in publish_release
return DockerClient().publish_release(source, target, deis_registry, creds)
File "/app/registry/dockerclient.py", line 114, in publish_release
self.push("{}/{}".format(self.registry, name), tag)
File "/usr/local/lib/python3.5/dist-packages/backoff/_sync.py", line 85, in retry
Build pod exited with code 1, stopping build.
sounds like something is gone wrong during your build, and you should read the output of the build logs to find out what...
You may be able to tail the build log through kubectl before the failed build pod is deleted... from the looks of it you're building Dockerfile, so the pod to tail during git push will be the dockerbuilder pod that builder creates
on builder it showing deployed complete .
Accepted connection.
Starting ssh authentication
Channel type: session
Key='LANG', Value='en_US.UTF-8'
receiving git repo name: artful-odometer.git, operation: git-receive-pack, fingerprint: 95:42:dd:24:2d:af:32:29:ed:40:2f:02:6b:7c:da:36, user: admin
creating repo directory /home/git/artful-odometer.git
writing pre-receive hook under /home/git/artful-odometer.git
git-shell -c git-receive-pack 'artful-odometer.git'
Waiting for git-receive to run.
Waiting for deploy.
Deploy complete.
It seems application container exited with error , I was trying git clone https://github.com/deis/example-go.git .
let me try with another one or I will try to write my own.
example-go is a Buildpack style, but example-dockerfile-http is a Dockerfile style
If you have switched from one such style of app to another, you will probably get unexpected behavior. This has been documented in previous issues on the deis repos but to my knowledge has never been completely or permanently solved, as there are easy workarounds.
So I'm thinking this is likely a bug you're running into if that's what you've done, but probably not a terribly interesting one for you to debug (unless you think you might be into that...)
To fix it, when you switch from one style app to another for debugging purposes or otherwise, please ensure that you're pushing your git repo or docker image to a new app (start with deis create).
There is some inference done by the builder/controller about what type of deployment it is, and certain transitions (like maybe dockerfile -> buildpack or vice versa) will cause these interesting types of failure. What you're doing if I have got it right where you've landed at now, has caused this inference to misbehave somehow and fail. Creating a new app will cause the inference to start from scratch and it should properly detect the type of app you're trying to build.
Please join us on the Deis slack where more users can help you debug this in real time; we should probably not be tutoring or live debugging/doing support on deployments of Deis on the github issues repo. I'd send you an invite to the Hephy slack but we're still launching and you're actually a lot more likely to get help on the Deis slack for now.
Don't mind the expired certificate
@yebyen
Thanks for detailed explanation and for slack workspace invite.
I solved my issue by restarting deis builder , now it's working fine.