Virtual desktop pods are not found + crashing KVDI manager
tbrodbeck opened this issue · comments
When I tried to run a template of a virtual desktop, I get these errors:
{
"error": "pods \"ubuntu-xfce-tndmb\" not found",
"status": "NotFound"
}{
"error": "pods \"dosbox-5hc9g\" not found",
"status": "NotFound"
}I then realised that the kvdi manager is crashing, which might be the reason for it (CrashLoopBackOff)
When I look into the logs I seepanic: runtime error: invalid memory address or nil pointer dereference
KVDI Manager Logs
2022-02-01T17:06:35.612Z INFO setup kVDI Version: v0.3.6
2022-02-01T17:06:35.612Z INFO setup kVDI Commit: 00f33f4eda3aef727dfbec324ef82865b656f873
2022-02-01T17:06:35.612Z INFO setup Go Version: go1.17.2
2022-02-01T17:06:35.612Z INFO setup Go OS/Arch: linux/amd64
2022-02-01T17:06:35.866Z INFO controller-runtime.metrics metrics server is starting to listen {"addr": "127.0.0.1:8080"}
2022-02-01T17:06:35.866Z INFO setup starting manager
I0201 17:06:35.867162 1 leaderelection.go:248] attempting to acquire leader lease kvdi/095fd8bb.kvdi.io...
2022-02-01T17:06:35.867Z INFO starting metrics server {"path": "/metrics"}
I0201 17:06:51.525120 1 leaderelection.go:258] successfully acquired lease kvdi/095fd8bb.kvdi.io
2022-02-01T17:06:51.525Z INFO controller.vdicluster Starting EventSource {"reconciler group": "app.kvdi.io", "reconciler kind": "VDICluster", "source": "kind source: /, Kind="}
2022-02-01T17:06:51.525Z INFO controller.session Starting EventSource {"reconciler group": "desktops.kvdi.io", "reconciler kind": "Session", "source": "kind source: /, Kind="}
2022-02-01T17:06:51.525Z INFO controller.vdicluster Starting EventSource {"reconciler group": "app.kvdi.io", "reconciler kind": "VDICluster", "source": "kind source: /, Kind="}
2022-02-01T17:06:51.525Z INFO controller.session Starting EventSource {"reconciler group": "desktops.kvdi.io", "reconciler kind": "Session", "source": "kind source: /, Kind="}
2022-02-01T17:06:51.525Z INFO controller.vdicluster Starting EventSource {"reconciler group": "app.kvdi.io", "reconciler kind": "VDICluster", "source": "kind source: /, Kind="}
2022-02-01T17:06:51.525Z INFO controller.session Starting EventSource {"reconciler group": "desktops.kvdi.io", "reconciler kind": "Session", "source": "kind source: /, Kind="}
2022-02-01T17:06:51.525Z INFO controller.vdicluster Starting EventSource {"reconciler group": "app.kvdi.io", "reconciler kind": "VDICluster", "source": "kind source: /, Kind="}
2022-02-01T17:06:51.525Z INFO controller.session Starting EventSource {"reconciler group": "desktops.kvdi.io", "reconciler kind": "Session", "source": "kind source: /, Kind="}
2022-02-01T17:06:51.525Z INFO controller.vdicluster Starting EventSource {"reconciler group": "app.kvdi.io", "reconciler kind": "VDICluster", "source": "kind source: /, Kind="}
2022-02-01T17:06:51.525Z INFO controller.session Starting EventSource {"reconciler group": "desktops.kvdi.io", "reconciler kind": "Session", "source": "kind source: /, Kind="}
2022-02-01T17:06:51.525Z INFO controller.vdicluster Starting EventSource {"reconciler group": "app.kvdi.io", "reconciler kind": "VDICluster", "source": "kind source: /, Kind="}
2022-02-01T17:06:51.525Z INFO controller.vdicluster Starting EventSource {"reconciler group": "app.kvdi.io", "reconciler kind": "VDICluster", "source": "kind source: /, Kind="}
2022-02-01T17:06:51.525Z INFO controller.vdicluster Starting Controller {"reconciler group": "app.kvdi.io", "reconciler kind": "VDICluster"}
2022-02-01T17:06:51.525Z INFO controller.session Starting Controller {"reconciler group": "desktops.kvdi.io", "reconciler kind": "Session"}
2022-02-01T17:06:51.525Z DEBUG events Normal {"object": {"kind":"ConfigMap","namespace":"kvdi","name":"095fd8bb.kvdi.io","uid":"f045eca8-6d6d-4fe3-95f5-3e189c200278","apiVersion":"v1","resourceVersion":"8126"}, "reason": "LeaderElection", "message": "kvdi-manager-9684bdc6b-mzfg5_d0de0aea-0439-4a4e-9ccd-a5af5afcc403 became leader"}
2022-02-01T17:06:51.525Z DEBUG events Normal {"object": {"kind":"Lease","namespace":"kvdi","name":"095fd8bb.kvdi.io","uid":"26e37f72-cb97-430b-8f43-ac33336f8f36","apiVersion":"coordination.k8s.io/v1","resourceVersion":"8127"}, "reason": "LeaderElection", "message": "kvdi-manager-9684bdc6b-mzfg5_d0de0aea-0439-4a4e-9ccd-a5af5afcc403 became leader"}
2022-02-01T17:06:51.629Z INFO controller.session Starting workers {"reconciler group": "desktops.kvdi.io", "reconciler kind": "Session", "worker count": 1}
2022-02-01T17:06:51.629Z INFO controller.vdicluster Starting workers {"reconciler group": "app.kvdi.io", "reconciler kind": "VDICluster", "worker count": 1}
2022-02-01T17:06:51.630Z INFO controllers.desktops.Session Reconciling Desktop {"session": "default/dosbox-5hc9g"}
2022-02-01T17:06:51.630Z INFO controllers.app.VDICluster Reconciling VDICluster {"vdicluster": "/kvdi"}
2022-02-01T17:06:51.630Z INFO controllers.desktops.Session Retrieving template and cluster for session {"session": "default/dosbox-5hc9g"}
2022-02-01T17:06:51.630Z INFO controllers.app.VDICluster Reconciling admin password secret {"vdicluster": "/kvdi"}
2022-02-01T17:06:51.630Z INFO controllers.app.VDICluster Setting up a temporary connection to the cluster secrets backend {"vdicluster": "/kvdi"}
2022-02-01T17:06:51.630Z INFO controllers.app.VDICluster Reconciling JWT secrets {"vdicluster": "/kvdi"}
2022-02-01T17:06:51.630Z INFO controllers.app.VDICluster Reconciling built-in VDIRoles {"vdicluster": "/kvdi"}
2022-02-01T17:06:51.630Z INFO controllers.app.VDICluster Reconciling required resources for the configured authentication provider {"vdicluster": "/kvdi"}
2022-02-01T17:06:51.630Z INFO controllers.app.VDICluster Reconciling RBAC resources for the app servers {"vdicluster": "/kvdi"}
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x13614de]
goroutine 373 [running]:
github.com/kvdi/kvdi/apis/app/v1.(*VDICluster).GetUserdataVolumeSpec(0xc00003d800)
/build/apis/app/v1/vdicluster_common_util.go:110 +0x5e
github.com/kvdi/kvdi/pkg/resources/desktop.(*Reconciler).Reconcile(0xc00017e2a0, {0x1a5f8a8, 0xc00017e180}, {0x1a7bfc0, 0xc0002d2240}, 0xc00044e180)
/build/pkg/resources/desktop/reconciler.go:94 +0x21a
github.com/kvdi/kvdi/controllers/desktops.(*SessionReconciler).Reconcile(0xc0007478f0, {0x1a5f8a8, 0xc00017e180}, {{{0xc0006e0ba9, 0x7}, {0xc0006e0b94, 0xc}}})
/build/controllers/desktops/session_controller.go:81 +0x2f5
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile(0xc000356b40, {0x1a5f8a8, 0xc00017e0f0}, {{{0xc0006e0ba9, 0x16fc440}, {0xc0006e0b94, 0xc00074f000}}})
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.10.2/pkg/internal/controller/controller.go:114 +0x222
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler(0xc000356b40, {0x1a5f800, 0xc00047b400}, {0x16813e0, 0xc000762000})
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.10.2/pkg/internal/controller/controller.go:311 +0x2f2
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem(0xc000356b40, {0x1a5f800, 0xc00047b400})
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.10.2/pkg/internal/controller/controller.go:266 +0x205
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2()
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.10.2/pkg/internal/controller/controller.go:227 +0x85
created by sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.10.2/pkg/internal/controller/controller.go:223 +0x354
Excerpt of App Logs
{"time":"2022-02-01T16:44:15.226781208Z","method":"GET","path":"/api/readyz","statusCode":200,"size":19,"remoteHost":"192.168.0.1"}
2022/02/01 16:44:22 http: TLS handshake error from 192.168.0.1:14731: remote error: tls: unknown certificate
2022/02/01 16:44:22 http: TLS handshake error from 192.168.0.1:40314: remote error: tls: unknown certificate
{"time":"2022-02-01T16:44:25.207026382Z","method":"GET","path":"/api/readyz","statusCode":200,"size":19,"remoteHost":"192.168.0.1"}
2022/02/01 16:44:25 http: TLS handshake error from 192.168.0.1:64067: remote error: tls: unknown certificate
2022/02/01 16:44:25 http: TLS handshake error from 192.168.0.1:22453: remote error: tls: unknown certificate
{"time":"2022-02-01T16:44:25.657811559Z","method":"GET","path":"/","statusCode":200,"size":900,"remoteHost":"192.168.0.1"}
{"time":"2022-02-01T16:44:25.751002852Z","method":"GET","path":"/css/app.983fa484.css","statusCode":200,"size":29063,"remoteHost":"192.168.0.1"}
{"time":"2022-02-01T16:44:25.750150202Z","method":"GET","path":"/css/vendor.954f525a.css","statusCode":200,"size":213477,"remoteHost":"192.168.0.1"}
{"time":"2022-02-01T16:44:25.751761858Z","method":"GET","path":"/js/app.343390e7.js","statusCode":200,"size":297062,"remoteHost":"192.168.0.1"}
{"time":"2022-02-01T16:44:25.751501852Z","method":"GET","path":"/js/vendor.3101e6dc.js","statusCode":200,"size":3481381,"remoteHost":"192.168.0.1"}
{"time":"2022-02-01T16:44:26.38269946Z","method":"GET","path":"/statics/icons/quasar.svg","statusCode":200,"size":2522,"remoteHost":"192.168.0.1"}
{"time":"2022-02-01T16:44:26.383207019Z","method":"GET","path":"/statics/github.png","statusCode":200,"size":24525,"remoteHost":"192.168.0.1"}
2022-02-01T16:44:26.387Z ERROR api Authentication failed, checking if anonymous is allowed{"error": "User 'anonymous' not found in the cluster"}
github.com/kvdi/kvdi/pkg/api.DecodeRequest.func1
/build/pkg/api/api_decoder.go:81
Forbidden request due to: User 'anonymous' not found in the cluster
net/http.HandlerFunc.ServeHTTP
/usr/local/go/src/net/http/server.go:2046
github.com/kvdi/kvdi/pkg/api.doRequestMetrics
/build/pkg/api/api_metrics.go:171
github.com/kvdi/kvdi/pkg/api.prometheusMiddleware.func1
/build/pkg/api/api_metrics.go:157
net/http.HandlerFunc.ServeHTTP
/usr/local/go/src/net/http/server.go:2046
github.com/gorilla/mux.(*Router).ServeHTTP
/go/pkg/mod/github.com/gorilla/mux@v1.8.0/mux.go:210
github.com/kvdi/kvdi/pkg/api.(*desktopAPI).ServeHTTP
/build/pkg/api/api_router.go:163
github.com/gorilla/mux.(*Router).ServeHTTP
/go/pkg/mod/github.com/gorilla/mux@v1.8.0/mux.go:210
github.com/gorilla/handlers.loggingHandler.ServeHTTP
/go/pkg/mod/github.com/gorilla/handlers@v1.5.1/logging.go:47
github.com/gorilla/handlers.CompressHandlerLevel.func1
/go/pkg/mod/github.com/gorilla/handlers@v1.5.1/compress.go:141
net/http.HandlerFunc.ServeHTTP
/usr/local/go/src/net/http/server.go:2046
github.com/gorilla/handlers.ProxyHeaders.func1
/go/pkg/mod/github.com/gorilla/handlers@v1.5.1/proxy_headers.go:59
net/http.HandlerFunc.ServeHTTP
/usr/local/go/src/net/http/server.go:2046
net/http.serverHandler.ServeHTTP
/usr/local/go/src/net/http/server.go:2878
net/http.initALPNRequest.ServeHTTP
/usr/local/go/src/net/http/server.go:3479
net/http.(*http2serverConn).runHandler
/usr/local/go/src/net/http/h2_bundle.go:5832
{"time":"2022-02-01T16:44:26.38826115Z","method":"GET","path":"/statics/app-logo-128x128.png","statusCode":200,"size":9181,"remoteHost":"192.168.0.1"}
{"time":"2022-02-01T16:44:26.383150054Z","method":"POST","path":"/api/login","statusCode":403,"size":77,"remoteHost":"192.168.0.1"}
{"time":"2022-02-01T16:44:26.432480528Z","method":"GET","path":"/fonts/KFOmCnqEu92Fr1Mu4mxM.9b78ea3b.woff","statusCode":200,"size":20332,"remoteHost":"192.168.0.1"}
{"time":"2022-02-01T16:44:26.432658237Z","method":"GET","path":"/fonts/KFOlCnqEu92Fr1MmEU9fBBc-.ddd11dab.woff","statusCode":200,"size":20532,"remoteHost":"192.168.0.1"}
{"time":"2022-02-01T16:44:26.432749242Z","method":"GET","path":"/fonts/KFOlCnqEu92Fr1MmWUlfBBc-.0344cc3c.woff","statusCode":200,"size":20396,"remoteHost":"192.168.0.1"}
{"time":"2022-02-01T16:44:26.427174318Z","method":"GET","path":"/fonts/flUhRq6tzZclQEJ-Vdg-IuiaDsNcIhQ8tQ.12730e02.woff2","statusCode":200,"size":113328,"remoteHost":"192.168.0.1"}
{"time":"2022-02-01T16:44:26.529928368Z","method":"GET","path":"/statics/icons/favicon.ico","statusCode":200,"size":5238,"remoteHost":"192.168.0.1"}
{"time":"2022-02-01T16:44:35.225822138Z","method":"GET","path":"/api/readyz","statusCode":200,"size":19,"remoteHost":"192.168.0.1"}
System:
KVDI Installation: helm install kvdi kvdi/kvdi -n kvdi --create-namespace
kubectl version --short
Client Version: v1.23.2
Server Version: v1.23.3
Docker Version: 20.10.12
Debian 11
Thanks in advance for any help!
I got the same error too and I think it has something to do with this line.
https://github.com/kvdi/kvdi/blob/c911719ffafb4f8c36ca1c17ed9480ef8f3ee1fb/apis/app/v1/vdicluster_common_util.go#L110
I think that c.Spec.UserdataSpec.PersistentVolumeClaimSpec is null, so the manager attempts to redeference a null value which causes an error. I'm not sure how to fix this in the code but I found a workaround by setting values in the PersistentVolumeClaimSpec for the VDICluster.
I created this yaml configuration in volume.yaml:
vdi:
spec:
userdataSpec:
accessModes:
- ReadWriteMany
resources:
requests:
storage: 1Gi
retainPVCs: trueand then ran helm install kvdi kvdi/kvdi -f volume.yaml and the error was fixed.
I hope this helps!
Thanks for the input. But I keep having the same problem 😭
After deleting and reinstalling the helm chart kvdi keeps crashing:
kubectl get all -n kvdi
NAME READY STATUS RESTARTS AGE
pod/kvdi-manager-9684bdc6b-ns478 1/2 CrashLoopBackOff 5 (2m49s ago) 7m26s
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/kvdi-manager-metrics-service ClusterIP 10.97.101.183 <none> 8443/TCP 7m26s
NAME READY UP-TO-DATE AVAILABLE AGE
deployment.apps/kvdi-manager 0/1 1 0 7m26s
NAME DESIRED CURRENT READY AGE
replicaset.apps/kvdi-manager-9684bdc6b 1 1 0 7m26s
Same error in the manager pod:
kubectl logs -n kvdi kvdi-manager-9684bdc6b-ns478 manager
[...]
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x13614de]
[...]
I cannot even start the web server. This is odd, the uninstalling the helm chart should wipe clean everything
Okay I was able to fix the crashing manager somehow when deleting this crds - helm does not do that by default for some reason:
kubectl delete crd sessions.desktops.kvdi.io templates.desktops.kvdi.io vdiclusters.app.kvdi.io vdiroles.rbac.kvdi.io
But I still have the problem of the missing pod (with or without the volume.yaml):
{
"error": "pods \"ubuntu-xfce-tndmb\" not found",
"status": "NotFound"
}
Huh, that's weird.
I would recommend deleting and re-adding the helm repo after uninstalling so the CRDs update. Also, creating a separate namespace for kvdi could be the issue. I think you have to specify which namespace to launch in when you start the connection. Here are the exact commands I ran (I'm using minikube by the way)
minikube start
helm repo remove kvdi
helm repo add kvdi https://kvdi.github.io/helm-charts/charts
helm repo update
helm install kvdi kvdi/kvdi -f volume.yaml
kubectl apply -f deploy/examples/example-desktop-templates.yaml
# Get base64 admin password
kubectl get secret kvdi-admin-secret -o go-template="{{ .data.password }}"
# Run this in seperate terminal
kubectl port-forward svc/kvdi-app 8443:443
# Access https://localhost:8443, login, and connect to ubuntu-xfceMake sure to watch the logs of the kvdi-manager for any errors.
If everything works correctly, you should see a ubuntu-xfce-***** pod created when you connect (run kubectl get pods). Just a warning though, the first run can take upwards of 15 minutes to start.
Hmm. Did that. But I am still getting the same errors (segmentation violation and "status": "NotFound")
I would check the configuration of the VDICluster to make sure the userdataSpec is set, by running kubectl describe vdicluster
You should get something like
Userdata Spec:
Access Modes:
ReadWriteOnce
Resources:
Requests:
Storage: 1Gi
Retain PVCs: true
at the bottom of config.
Also, running minikube delete before starting might help you too. If this doesn't work, then I'm not sure what else could be different😭
Hey! First of all, I am not using minikube. I am using a bare metal cluster on debian 11.
Ok I did that:
➜ k8s git:(main) ✗ kubectl describe vdicluster
Name: kvdi
Namespace:
Labels: app.kubernetes.io/managed-by=Helm
component=kvdi-cluster
Annotations: meta.helm.sh/release-name: kvdi
meta.helm.sh/release-namespace: kvdi
API Version: app.kvdi.io/v1
Kind: VDICluster
Metadata:
Creation Timestamp: 2022-02-09T08:44:45Z
Generation: 1
Managed Fields:
API Version: app.kvdi.io/v1
Fields Type: FieldsV1
fieldsV1:
f:metadata:
f:annotations:
.:
f:meta.helm.sh/release-name:
f:meta.helm.sh/release-namespace:
f:labels:
.:
f:app.kubernetes.io/managed-by:
f:component:
f:spec:
.:
f:app:
.:
f:auditLog:
f:corsEnabled:
f:image:
f:replicas:
f:resources:
f:serviceAnnotations:
f:serviceType:
f:tls:
.:
f:serverSecret:
f:appNamespace:
f:auth:
.:
f:adminSecret:
f:allowAnonymous:
f:ldapAuth:
f:localAuth:
f:oidcAuth:
f:tokenDuration:
f:desktops:
.:
f:maxSessionLength:
f:imagePullSecrets:
f:metrics:
.:
f:serviceMonitor:
.:
f:create:
f:labels:
.:
f:release:
f:secrets:
.:
f:k8sSecret:
.:
f:secretName:
f:vault:
f:userdataSpec:
Manager: helm
Operation: Update
Time: 2022-02-09T08:44:45Z
Resource Version: 772554
UID: cafb75a7-3377-43ce-8dac-a6ff5922d1a2
Spec:
App:
Audit Log: false
Cors Enabled: false
Image:
Replicas: 1
Resources:
Service Annotations:
Service Type: LoadBalancer
Tls:
Server Secret:
App Namespace: default
Auth:
Admin Secret: kvdi-admin-secret
Allow Anonymous: false
Ldap Auth:
Local Auth:
Oidc Auth:
Token Duration: 15m
Desktops:
Max Session Length:
Image Pull Secrets:
Metrics:
Service Monitor:
Create: false
Labels:
Release: prometheus
Secrets:
k8sSecret:
Secret Name: kvdi-app-secrets
Vault:
Userdata Spec:
Events: <none>
You are right. My Userdata Spec are empty instead
If you can set the Userdata Spec field then the bug should be fixed. Adding -f volume.yaml should set it, so make sure you have tried the yaml file from earlier. You could also try setting a field manually through adding
--set vdi.spec.userdataSpec.resources.requests.storage=1Gi
Ok sorry my bad. The file was empty for some reason. Thanks for pointing that out.
Now I get
Userdata Spec:
Access Modes:
ReadWriteMany
Resources:
Requests:
Storage: 1Gi
Retain PV Cs: true
Now I can start the template, but it stays on pending:
pod/ubuntu-xfce-gsqdg 0/3 Pending 0 105m
There is nothing about that in the logs, the best I could find is this from the manager:
2022-02-09T14:31:42.287Z INFO controllers.desktops.Session Reconciling Desktop {"session": "default/ubuntu-xfce-gsqdg"}
2022-02-09T14:31:42.287Z INFO controllers.desktops.Session Retrieving template and cluster for session {"session": "default/ubuntu-xfce-gsqdg"}
2022-02-09T14:31:42.287Z INFO controllers.desktops.Session Cluster has userdataSpec, reconciling volumes {"session": "default/ubuntu-xfce-gsqdg"}
2022-02-09T14:31:42.287Z INFO controllers.desktops.Session Reconciling service for the desktop session {"session": "default/ubuntu-xfce-gsqdg"}
2022-02-09T14:31:42.287Z INFO controllers.desktops.Session Generating mTLS certificate for the session proxy {"session": "default/ubuntu-xfce-gsqdg"}
2022-02-09T14:31:42.288Z INFO controllers.desktops.Session Reconciling pod for session {"session": "default/ubuntu-xfce-gsqdg"}
2022-02-09T14:31:42.296Z INFO controllers.desktops.Session Requeueing in 3 seconds for: Desktop pod is not in running phase {"session": "default/ubuntu-xfce-gsqdg"}
Ah, ran into that issue too. It stays on pending because are not enough resources available to start the pod.
If you run kubectl describe nodes you should see that there not enough resources for your nodes. I'm not sure how to expand resources on your setup, but the nodes likely need more CPU or memory.
Hmm thats wierd. Check this output. Looks like the node does have enough capacity (its only at 15% load):
➜ k8s git:(main) ✗ kubectl get all
NAME READY STATUS RESTARTS AGE
pod/jupyter-848ffcf4b7-gmf56 1/1 Running 2 (17h ago) 7d
pod/kvdi-app-66f558968b-hrfdx 1/1 Running 0 6m28s
pod/kvdi-manager-9684bdc6b-p6rxx 2/2 Running 0 6m49s
pod/ubuntu-xfce-gsqdg 0/3 Pending 0 44s
pod/ubuntu-xfce-qlqld 0/3 Pending 0 3m30s
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/jupyter ClusterIP 10.101.180.11 <none> 8888/TCP 7d
service/kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 7d1h
service/kvdi-app LoadBalancer 10.96.103.226 192.168.178.241 443:30322/TCP 6m24s
service/kvdi-manager-metrics-service ClusterIP 10.101.88.175 <none> 8443/TCP 6m49s
service/ubuntu-xfce-gsqdg ClusterIP 10.97.102.144 <none> 8443/TCP 2m13s
service/ubuntu-xfce-qlqld ClusterIP 10.96.76.193 <none> 8443/TCP 3m34s
NAME READY UP-TO-DATE AVAILABLE AGE
deployment.apps/jupyter 1/1 1 1 7d
deployment.apps/kvdi-app 1/1 1 1 6m28s
deployment.apps/kvdi-manager 1/1 1 1 6m49s
NAME DESIRED CURRENT READY AGE
replicaset.apps/jupyter-848ffcf4b7 1 1 1 7d
replicaset.apps/kvdi-app-66f558968b 1 1 1 6m28s
replicaset.apps/kvdi-manager-9684bdc6b 1 1 1 6m49s
➜ k8s git:(main) ✗ kubectl describe nodes
Name: node1
Roles: control-plane,master
Labels: beta.kubernetes.io/arch=amd64
beta.kubernetes.io/os=linux
kubernetes.io/arch=amd64
kubernetes.io/hostname=node1
kubernetes.io/os=linux
node-role.kubernetes.io/control-plane=
node-role.kubernetes.io/master=
node.kubernetes.io/exclude-from-external-load-balancers=
Annotations: kubeadm.alpha.kubernetes.io/cri-socket: /var/run/dockershim.sock
node.alpha.kubernetes.io/ttl: 0
projectcalico.org/IPv4Address: 192.168.178.200/24
projectcalico.org/IPv4IPIPTunnelAddr: 172.16.166.128
volumes.kubernetes.io/controller-managed-attach-detach: true
CreationTimestamp: Wed, 02 Feb 2022 17:10:23 +0100
Taints: <none>
Unschedulable: false
Lease:
HolderIdentity: node1
AcquireTime: <unset>
RenewTime: Wed, 09 Feb 2022 18:13:39 +0100
Conditions:
Type Status LastHeartbeatTime LastTransitionTime Reason Message
---- ------ ----------------- ------------------ ------ -------
NetworkUnavailable False Wed, 09 Feb 2022 00:19:01 +0100 Wed, 09 Feb 2022 00:19:01 +0100 CalicoIsUp Calico is running on this node
MemoryPressure False Wed, 09 Feb 2022 18:12:21 +0100 Wed, 02 Feb 2022 17:10:23 +0100 KubeletHasSufficientMemory kubelet has sufficient memory available
DiskPressure False Wed, 09 Feb 2022 18:12:21 +0100 Wed, 02 Feb 2022 17:10:23 +0100 KubeletHasNoDiskPressure kubelet has no disk pressure
PIDPressure False Wed, 09 Feb 2022 18:12:21 +0100 Wed, 02 Feb 2022 17:10:23 +0100 KubeletHasSufficientPID kubelet has sufficient PID available
Ready True Wed, 09 Feb 2022 18:12:21 +0100 Wed, 02 Feb 2022 17:14:45 +0100 KubeletReady kubelet is posting ready status. AppArmor enabled
Addresses:
InternalIP: 192.168.178.200
Hostname: node1
Capacity:
cpu: 8
ephemeral-storage: 121499304Ki
hugepages-2Mi: 0
memory: 16285328Ki
pods: 110
Allocatable:
cpu: 8
ephemeral-storage: 111973758382
hugepages-2Mi: 0
memory: 16182928Ki
pods: 110
System Info:
Machine ID: f22d090aacd04704b81aa94ee5d6ae30
System UUID: 4c4c4544-0054-4210-8036-b6c04f44354a
Boot ID: 721b5765-6542-4394-878e-b87e8a1b5d40
Kernel Version: 5.10.0-10-amd64
OS Image: Debian GNU/Linux 11 (bullseye)
Operating System: linux
Architecture: amd64
Container Runtime Version: docker://20.10.12
Kubelet Version: v1.23.1
Kube-Proxy Version: v1.23.1
Non-terminated Pods: (15 in total)
Namespace Name CPU Requests CPU Limits Memory Requests Memory Limits Age
--------- ---- ------------ ---------- --------------- ------------- ---
default jupyter-848ffcf4b7-gmf56 0 (0%) 0 (0%) 0 (0%) 0 (0%) 7d
default kvdi-app-66f558968b-hrfdx 0 (0%) 0 (0%) 0 (0%) 0 (0%) 6m35s
default kvdi-manager-9684bdc6b-p6rxx 0 (0%) 0 (0%) 0 (0%) 0 (0%) 6m56s
ingress-nginx-system ingress-nginx-controller-5sjgh 100m (1%) 0 (0%) 90Mi (0%) 0 (0%) 7d
kube-system calico-kube-controllers-566dc76669-n6x2t 0 (0%) 0 (0%) 0 (0%) 0 (0%) 7d
kube-system calico-node-thhcv 250m (3%) 0 (0%) 0 (0%) 0 (0%) 7d
kube-system coredns-64897985d-h64mj 100m (1%) 0 (0%) 70Mi (0%) 170Mi (1%) 7d1h
kube-system coredns-64897985d-pgfwd 100m (1%) 0 (0%) 70Mi (0%) 170Mi (1%) 7d1h
kube-system etcd-node1 100m (1%) 0 (0%) 100Mi (0%) 0 (0%) 7d1h
kube-system kube-apiserver-node1 250m (3%) 0 (0%) 0 (0%) 0 (0%) 7d1h
kube-system kube-controller-manager-node1 200m (2%) 0 (0%) 0 (0%) 0 (0%) 7d1h
kube-system kube-proxy-crvsx 0 (0%) 0 (0%) 0 (0%) 0 (0%) 7d1h
kube-system kube-scheduler-node1 100m (1%) 0 (0%) 0 (0%) 0 (0%) 7d1h
metallb-system controller-7cf77c64fb-sxmb8 0 (0%) 0 (0%) 0 (0%) 0 (0%) 7d
metallb-system speaker-pp2pg 0 (0%) 0 (0%) 0 (0%) 0 (0%) 7d
Allocated resources:
(Total limits may be over 100 percent, i.e., overcommitted.)
Resource Requests Limits
-------- -------- ------
cpu 1200m (15%) 0 (0%)
memory 330Mi (2%) 340Mi (2%)
ephemeral-storage 0 (0%) 0 (0%)
hugepages-2Mi 0 (0%) 0 (0%)
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning listen tcp4 :30322: bind: address already in use 6m30s kube-proxy can't open port "nodePort for default/kvdi-app:web" (:30322/tcp4), skipping it
I think you don't have enough memory for the ubuntu-xfce pod.
kubectl describe nodes says you have are using 330Mi but have a limit of 340Mi.
I think the limit of 340Mi refer to the two requests from coredns (2x 170Mi). This does not refer to the overall capacity of the node. Actually, this node has 16GB of RAM and I did not set any limits there, so this should not be the concern. 🤔
I found that there is a not working pvc:
➜ k8s git:(main) ✗ k describe pod ubuntu-xfce-nxx98
[...]
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning FailedScheduling 34s default-scheduler 0/1 nodes are available: 1 pod has unbound immediate PersistentVolumeClaims.
➜ k8s git:(main) ✗ k describe pvc kvdi-admin-userdata
Name: kvdi-admin-userdata
Namespace: default
StorageClass:
Status: Pending
Volume:
Labels: desktopName=ubuntu-xfce-ndgfd
desktopUser=admin
vdiCluster=kvdi
vdiComponent=desktop
Annotations: kvdi.io/creation-spec: 9e78d334dfe5e2f03c42f5316d9f6f8b9fd69d86d735991e00804a559cca2c77
Finalizers: [kubernetes.io/pvc-protection]
Capacity:
Access Modes:
VolumeMode: Filesystem
Used By: ubuntu-xfce-nxx98
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal FailedBinding 112s (x5002 over 20h) persistentvolume-controller no persistent volumes available for this claim and no storage class is set
Sorry, my bad, you are right. minikube automatically creates everything for me, so I don't have to worry about managing persistent volumes. You could follow the advice set out here https://stackoverflow.com/questions/55780083/error-no-persistent-volumes-available-for-this-claim-and-no-storage-class-is-se or here https://kubernetes.io/docs/concepts/storage/dynamic-provisioning to fix it.
I was having this issue on k3s.
I followed @just-luk advice for using a volume.yaml
That resolved my crashing issue but the vdi pod was then stuck in pending.
To resolve the pending issue I needed to change the Access Modes: to ReadWriteOnce
This setting will depend on what kind of storage you are using but for my default k3s install this is what worked for me
Hey. Thanks, @just-luk I tried to change the storage class of the pvc that kvdi creates, but it is not possible:
The PersistentVolumeClaim "kvdi-admin-userdata" is invalid: spec: Forbidden: spec is immutable after creation except resources.requests for bound claims
core.PersistentVolumeClaimSpec{
... // 2 identical fields
Resources: {Requests: {s"storage": {i: {...}, s: "1Gi", Format: "BinarySI"}}},
VolumeName: "",
- StorageClassName: &"standard",
+ StorageClassName: nil,
VolumeMode: &"Filesystem",
DataSource: nil,
DataSourceRef: nil,
}
Then, I tried to create a pvc without a storage class name, but it is ignored by kvdi nevertheless:
➜ ~ kubectl get pv
NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE
jupyter 10Gi RWX Retain Bound default/jupyter standard 11d
kvdi 10Gi RWX Retain Available 37
The solution for Helm pointed by @just-luk worked for me, however on GKE I had yet another error after that. It was related to missing the accessMode attribute for PVC requests. The Helm command below handled both:
helm upgrade --install
--set vdi.spec.userdataSpec.resources.requests.storage=10Gi
--set vdi.spec.userdataSpec.accessModes={ReadWriteOnce}
kvdi kvdi/kvdi