[Linux: Docker Desktop] - Installation stuck on "Deploying Orchest Control Plane"

Question

[Linux: Docker Desktop] - Installation stuck on "Deploying Orchest Control Plane"

nuhyurdev opened this issue 2 years ago · comments

Hello, I have a problem to install orchest and to deploy one cluster kubernetes env. I used docker desktop and enabled kubernetes. I am trying to install orchest my server via the code below:

.local/bin/orchest install

but it's stuck on deploying control plane.All containers running in docker desktop but it's not happened anything for 2 hour. the installation script was not terminated.
How can I find a solutions for this ?

Yannick Perrenet · Answer 1 · Mon Oct 10 2022 18:45:40 GMT+0800 (China Standard Time)

Hi @nuhyurduseven, I am sorry to hear that. Let's get the issue resolved 😸

To rule out certain issues, could you try visiting http://localhost:80 to see whether that displays the Orchest UI for you?

Based on your screenshot it seems like the status of the underlying OrchestCluster was never put to RUNNING, which results in the orchest-cli never stopping to output the Deploying Orchest Control Plane. Would you mind sharing the following:

orchest-controller logs: Output of kubectl -n orchest logs -l app.kubernetes.io/name=orchest-controller
OrchestCluster description: Output of kubectl -n orchest describe orchestcluster cluster-1
What OS are you running on? I guess (based on #1349) Ubuntu 20.04?

(From the Docker docs it seems like kubectl is not shipped by default on Linux, so you would have to install it first: https://kubernetes.io/docs/tasks/tools/install-kubectl-linux/.)

Thank you.

nuh yurduseven · Answer 2 · Mon Oct 10 2022 19:37:55 GMT+0800 (China Standard Time)

Hi, now I try to access localhost:80 address but connection refused.

>  ~ curl -G  http://localhost:80                                                 
curl: (7) Failed to connect to localhost port 80: Connection refused

I found the error orchest-controller logs

Problem is absolutely here.

this is the output of kubectl -n orchest logs -l app.kubernetes.io/name=orchest-controller
E1010 10:40:06.535042 1 controller.go:172] failed to add finalizer "controller.orchest.io" to "orchest-database": Operation cannot be fulfilled on orchestcomponents.orchest.io "orchest-database": the object has been modified; please apply your changes to the latest version and try again
this is the output of kubectl -n orchest describe orchestcluster cluster-1
Name: cluster-1
Namespace: orchest
Labels:
Annotations: controller.orchest.io/k8s: docker-desktop
API Version: orchest.io/v1alpha1
Kind: OrchestCluster
Metadata:
Creation Timestamp: 2022-10-10T10:39:36Z
Finalizers:
controller.orchest.io
Generation: 3
Managed Fields:
API Version: orchest.io/v1alpha1
Fields Type: FieldsV1
fieldsV1:
f:spec:
.:
f:orchest:
.:
f:authServer:
f:orchestWebServer:
f:resources:
.:
f:userDirVolumeSize:
Manager: OpenAPI-Generator
Operation: Update
Time: 2022-10-10T10:39:36Z
API Version: orchest.io/v1alpha1
Fields Type: FieldsV1
fieldsV1:
f:metadata:
f:annotations:
.:
f:controller.orchest.io/k8s:
f:finalizers:
.:
v:"controller.orchest.io":
f:spec:
f:applications:
f:orchest:
f:authServer:
f:env:
f:image:
f:buildkitDaemon:
.:
f:image:
f:celeryWorker:
.:
f:env:
f:image:
f:env:
f:nodeAgent:
.:
f:image:
f:orchestApi:
.:
f:env:
f:image:
f:orchestWebServer:
f:env:
f:image:
f:pause:
f:version:
f:postgres:
.:
f:env:
f:image:
f:rabbitMq:
.:
f:image:
f:singleNode:
Manager: controller
Operation: Update
Time: 2022-10-10T10:39:39Z
API Version: orchest.io/v1alpha1
Fields Type: FieldsV1
fieldsV1:
f:status:
.:
f:conditions:
f:lastHeartbeatTime:
f:observedHash:
f:state:
Manager: controller
Operation: Update
Subresource: status
Time: 2022-10-10T10:41:31Z
Resource Version: 16754
UID: 628bf2dd-beb6-48a8-9a74-9e614efa69d9
Spec:
Applications:
Config:
Helm:
Parameters:
Name: persistence.size
Value: 25Gi
Name: service.clusterIP
Value: 10.96.0.2
Name: docker-registry
Config:
Helm:
Parameters:
Name: singleNamespace
Value: true
Name: argo-workflow
Orchest:
Auth Server:
Env:
Name: CLOUD
Value: False
Name: FLASK_ENV
Value: production
Image: orchest/auth-server:v2022.10.2
Buildkit Daemon:
Image: orchest/buildkit-daemon:v2022.10.2
Celery Worker:
Env:
Name: ORCHEST_GPU_ENABLED_INSTANCE
Value: FALSE
Name: FLASK_ENV
Value: production
Name: SINGLE_NODE
Value: TRUE
Image: orchest/celery-worker:v2022.10.2
Env:
Name: CONTAINER_RUNTIME_SOCKET
Value: /var/run/docker.sock
Name: CONTAINER_RUNTIME_IMAGE
Value: orchest/image-puller:v2022.10.2
Name: K8S_DISTRO
Value: docker-desktop
Name: PYTHONUNBUFFERED
Value: TRUE
Name: ORCHEST_LOG_LEVEL
Value: INFO
Name: ORCHEST_HOST_GID
Value: 1
Name: CONTAINER_RUNTIME
Value: docker
Name: ORCHEST_CLUSTER
Value: cluster-1
Name: ORCHEST_NAMESPACE
Value: orchest
Node Agent:
Image: orchest/node-agent:v2022.10.2
Orchest API:
Env:
Name: ORCHEST_GPU_ENABLED_INSTANCE
Value: FALSE
Name: FLASK_ENV
Value: production
Name: SINGLE_NODE
Value: TRUE
Image: orchest/orchest-api:v2022.10.2
Orchest Web Server:
Env:
Name: CLOUD
Value: False
Name: HOST_CONFIG_DIR
Value: /var/lib/orchest/config
Name: HOST_REPO_DIR
Value: /var/lib/orchest/repo
Name: HOST_OS
Value: linux
Name: ORCHEST_GPU_ENABLED_INSTANCE
Value: FALSE
Name: FLASK_ENV
Value: production
Name: ORCHEST_PORT
Value: 8000
Name: USERDIR_PVC
Value: userdir-pvc
Image: orchest/orchest-webserver:v2022.10.2
Pause: false
Resources:
User Dir Volume Size: 50Gi
Version: v2022.10.2
Postgres:
Env:
Name: PGDATA
Value: /userdir/.orchest/database/data
Name: POSTGRES_HOST_AUTH_METHOD
Value: trust
Image: postgres:13.1
Rabbit Mq:
Image: rabbitmq:3
Single Node: true
Status:
Conditions:
Event: deploying orchest-database
Last Heartbeat Time: 2022-10-10T10:40:06Z
Last Transition Time: 2022-10-10T10:40:06Z
Event: deploying rabbitmq-server
Last Heartbeat Time: 2022-10-10T10:40:20Z
Last Transition Time: 2022-10-10T10:40:20Z
Event: deploying orchest-api
Last Heartbeat Time: 2022-10-10T10:40:29Z
Last Transition Time: 2022-10-10T10:40:29Z
Event: deploying celery-worker
Last Heartbeat Time: 2022-10-10T10:41:09Z
Last Transition Time: 2022-10-10T10:41:09Z
Event: deploying auth-server
Last Heartbeat Time: 2022-10-10T10:41:20Z
Last Transition Time: 2022-10-10T10:41:20Z
Event: deploying orchest-webserver
Last Heartbeat Time: 2022-10-10T10:41:31Z
Last Transition Time: 2022-10-10T10:41:31Z
Last Heartbeat Time: 2022-10-10T10:40:05Z
Observed Hash: 85d6b65fb6
State: Deploying Orchest Control Plane
Events:
and I use Ubuntu 20.04 focal fossa.
additionally I tried to install with minikube and got error below in output of script

Error making standard the default storage class: Error listing StorageClasses: Get "https://192.168.49.2:8443/apis/storage.k8s.io/v1/storageclasses": dial tcp 192.168.49.2:8443: i/o timeout

nuh yurduseven · Answer 3 · Mon Oct 10 2022 19:49:03 GMT+0800 (China Standard Time)

Additionally I got the error during installation with minikube below

Configuring RBAC rules ...- E1010 11:28:02.876277 4334 start.go:268] Unable to scale down deployment "coredns" in namespace "kube-system" to 1 replica: timed out waiting for the condition

nuh yurduseven · Answer 4 · Mon Oct 10 2022 22:41:04 GMT+0800 (China Standard Time)

I uninstalled docker-desktop,minikube, kubernetes, deleted all images and reinstall using convenience_install script via minikube install way. It's done. I can access on firefox browser using minikube ip. I can use this app. But I think there is something wrong about docker desktop installation this app.

Yannick Perrenet · Answer 5 · Tue Oct 11 2022 15:48:39 GMT+0800 (China Standard Time)

Thank you for all that effort @nuhyurduseven ❤️. Happy to hear that you got Orchest running.

I will try to reproduce the errors on Docker Desktop this week. Let's keep the issue open to track progress.

EDIT: I tried to install Docker Desktop on Linux but sadly my system appeared to be in a state in which I was unable to install Docker Desktop following their docs. Moreover, their docs mention that it is not supported to run Docker Desktop on Linux in a virtualized Linux environment (so I couldn't do that either). Thus, for now, I will leave the issue open until someone has the time to try and reproduce.

Rick Lamers · Answer 6 · Wed Oct 26 2022 00:26:14 GMT+0800 (China Standard Time)

We've deprecated Docker for Desktop based installations. For Linux only minikube is supported as of now. Glad @nuhyurduseven was able to get that to work!