Deploying cluster fails waiting for kubernetes to be ready
pixil98 opened this issue · comments
I've been using this provider to deploy kubernetes clusters on top of Proxmox VMs for a while now. Recently I tried deploying a cluster to do some testing and ran into the following error and now I can't seem to get a cluster to deploy. I've tried on both version 1.4.2 and 1.4.3, same error.
Any ideas on what is going wrong or next steps for troubleshooting?
│ Error:
│ ============= RKE outputs ==============
│ time="2023-12-30T22:16:24Z" level=info msg="[rke_provider] rke cluster changed arguments: map[addons:true ingress:true kubernetes_version:true nodes:true private_registries:true]"
│ time="2023-12-30T22:16:24Z" level=info msg="Creating RKE cluster..."
│ time="2023-12-30T22:16:24Z" level=info msg="Initiating Kubernetes cluster"
│ time="2023-12-30T22:16:24Z" level=info msg="[dialer] Setup tunnel for host [192.168.1.41]"
│ time="2023-12-30T22:16:24Z" level=info msg="[dialer] Setup tunnel for host [192.168.1.40]"
│ time="2023-12-30T22:16:24Z" level=info msg="Finding container [cluster-state-deployer] on host [192.168.1.40], try #1"
│ time="2023-12-30T22:16:24Z" level=info msg="Pulling image [registry.lab.reisman.org/proxy.docker.io/rancher/rke-tools:v0.1.96] on host [192.168.1.40], try #1"
│ time="2023-12-30T22:16:39Z" level=info msg="Image [registry.lab.reisman.org/proxy.docker.io/rancher/rke-tools:v0.1.96] exists on host [192.168.1.40]"
│ time="2023-12-30T22:16:41Z" level=info msg="Starting container [cluster-state-deployer] on host [192.168.1.40], try #1"
│ time="2023-12-30T22:16:41Z" level=info msg="[state] Successfully started [cluster-state-deployer] container on host [192.168.1.40]"
│ time="2023-12-30T22:16:41Z" level=info msg="Finding container [cluster-state-deployer] on host [192.168.1.41], try #1"
│ time="2023-12-30T22:16:41Z" level=info msg="Pulling image [registry.lab.reisman.org/proxy.docker.io/rancher/rke-tools:v0.1.96] on host [192.168.1.41], try #1"
│ time="2023-12-30T22:16:54Z" level=info msg="Image [registry.lab.reisman.org/proxy.docker.io/rancher/rke-tools:v0.1.96] exists on host [192.168.1.41]"
│ time="2023-12-30T22:16:55Z" level=info msg="Starting container [cluster-state-deployer] on host [192.168.1.41], try #1"
│ time="2023-12-30T22:16:56Z" level=info msg="[state] Successfully started [cluster-state-deployer] container on host [192.168.1.41]"
│ time="2023-12-30T22:16:56Z" level=info msg="[certificates] Generating CA kubernetes certificates"
│ time="2023-12-30T22:16:56Z" level=info msg="[certificates] Generating Kubernetes API server aggregation layer requestheader client CA certificates"
│ time="2023-12-30T22:16:57Z" level=info msg="[certificates] GenerateServingCertificate is disabled, checking if there are unused kubelet certificates"
│ time="2023-12-30T22:16:57Z" level=info msg="[certificates] Generating Kubernetes API server certificates"
│ time="2023-12-30T22:16:58Z" level=info msg="[certificates] Generating Service account token key"
│ time="2023-12-30T22:16:58Z" level=info msg="[certificates] Generating Kube Controller certificates"
│ time="2023-12-30T22:16:58Z" level=info msg="[certificates] Generating Kube Scheduler certificates"
│ time="2023-12-30T22:16:58Z" level=info msg="[certificates] Generating Kube Proxy certificates"
│ time="2023-12-30T22:16:59Z" level=info msg="[certificates] Generating Node certificate"
│ time="2023-12-30T22:16:59Z" level=info msg="[certificates] Generating admin certificates and kubeconfig"
│ time="2023-12-30T22:16:59Z" level=info msg="[certificates] Generating Kubernetes API server proxy client certificates"
│ time="2023-12-30T22:17:00Z" level=info msg="[certificates] Generating kube-etcd-192-168-1-40 certificate and key"
│ time="2023-12-30T22:17:01Z" level=info msg="Successfully Deployed state file at [/home/aaron/homelab-deployments/development/terraform-provider-rke-tmp-2841594254/cluster.rkestate]"
│ time="2023-12-30T22:17:01Z" level=info msg="Building Kubernetes cluster"
│ time="2023-12-30T22:17:01Z" level=info msg="[dialer] Setup tunnel for host [192.168.1.40]"
│ time="2023-12-30T22:17:01Z" level=info msg="[dialer] Setup tunnel for host [192.168.1.41]"
│ time="2023-12-30T22:17:01Z" level=info msg="[network] Deploying port listener containers"
│ time="2023-12-30T22:17:01Z" level=info msg="Image [registry.lab.reisman.org/proxy.docker.io/rancher/rke-tools:v0.1.96] exists on host [192.168.1.40]"
│ time="2023-12-30T22:17:02Z" level=info msg="Starting container [rke-etcd-port-listener] on host [192.168.1.40], try #1"
│ time="2023-12-30T22:17:02Z" level=info msg="[network] Successfully started [rke-etcd-port-listener] container on host [192.168.1.40]"
│ time="2023-12-30T22:17:02Z" level=info msg="Image [registry.lab.reisman.org/proxy.docker.io/rancher/rke-tools:v0.1.96] exists on host [192.168.1.40]"
│ time="2023-12-30T22:17:03Z" level=info msg="Starting container [rke-cp-port-listener] on host [192.168.1.40], try #1"
│ time="2023-12-30T22:17:04Z" level=info msg="[network] Successfully started [rke-cp-port-listener] container on host [192.168.1.40]"
│ time="2023-12-30T22:17:04Z" level=info msg="Image [registry.lab.reisman.org/proxy.docker.io/rancher/rke-tools:v0.1.96] exists on host [192.168.1.41]"
│ time="2023-12-30T22:17:04Z" level=info msg="Starting container [rke-worker-port-listener] on host [192.168.1.41], try #1"
│ time="2023-12-30T22:17:05Z" level=info msg="[network] Successfully started [rke-worker-port-listener] container on host [192.168.1.41]"
│ time="2023-12-30T22:17:05Z" level=info msg="[network] Port listener containers deployed successfully"
│ time="2023-12-30T22:17:05Z" level=info msg="[network] Running control plane -> etcd port checks"
│ time="2023-12-30T22:17:05Z" level=info msg="[network] Checking if host [192.168.1.40] can connect to host(s) [192.168.1.40] on port(s) [2379], try #1"
│ time="2023-12-30T22:17:05Z" level=info msg="Image [registry.lab.reisman.org/proxy.docker.io/rancher/rke-tools:v0.1.96] exists on host [192.168.1.40]"
│ time="2023-12-30T22:17:06Z" level=info msg="Starting container [rke-port-checker] on host [192.168.1.40], try #1"
│ time="2023-12-30T22:17:06Z" level=info msg="[network] Successfully started [rke-port-checker] container on host [192.168.1.40]"
│ time="2023-12-30T22:17:06Z" level=info msg="Removing container [rke-port-checker] on host [192.168.1.40], try #1"
│ time="2023-12-30T22:17:06Z" level=info msg="[network] Running control plane -> worker port checks"
│ time="2023-12-30T22:17:06Z" level=info msg="[network] Checking if host [192.168.1.40] can connect to host(s) [192.168.1.41] on port(s) [10250], try #1"
│ time="2023-12-30T22:17:06Z" level=info msg="Image [registry.lab.reisman.org/proxy.docker.io/rancher/rke-tools:v0.1.96] exists on host [192.168.1.40]"
│ time="2023-12-30T22:17:07Z" level=info msg="Starting container [rke-port-checker] on host [192.168.1.40], try #1"
│ time="2023-12-30T22:17:07Z" level=info msg="[network] Successfully started [rke-port-checker] container on host [192.168.1.40]"
│ time="2023-12-30T22:17:07Z" level=info msg="Removing container [rke-port-checker] on host [192.168.1.40], try #1"
│ time="2023-12-30T22:17:07Z" level=info msg="[network] Running workers -> control plane port checks"
│ time="2023-12-30T22:17:07Z" level=info msg="[network] Checking if host [192.168.1.41] can connect to host(s) [192.168.1.40] on port(s) [6443], try #1"
│ time="2023-12-30T22:17:07Z" level=info msg="Image [registry.lab.reisman.org/proxy.docker.io/rancher/rke-tools:v0.1.96] exists on host [192.168.1.41]"
│ time="2023-12-30T22:17:08Z" level=info msg="Starting container [rke-port-checker] on host [192.168.1.41], try #1"
│ time="2023-12-30T22:17:08Z" level=info msg="[network] Successfully started [rke-port-checker] container on host [192.168.1.41]"
│ time="2023-12-30T22:17:08Z" level=info msg="Removing container [rke-port-checker] on host [192.168.1.41], try #1"
│ time="2023-12-30T22:17:08Z" level=info msg="[network] Checking KubeAPI port Control Plane hosts"
│ time="2023-12-30T22:17:08Z" level=info msg="[network] Removing port listener containers"
│ time="2023-12-30T22:17:08Z" level=info msg="Removing container [rke-etcd-port-listener] on host [192.168.1.40], try #1"
│ time="2023-12-30T22:17:08Z" level=info msg="[remove/rke-etcd-port-listener] Successfully removed container on host [192.168.1.40]"
│ time="2023-12-30T22:17:08Z" level=info msg="Removing container [rke-cp-port-listener] on host [192.168.1.40], try #1"
│ time="2023-12-30T22:17:09Z" level=info msg="[remove/rke-cp-port-listener] Successfully removed container on host [192.168.1.40]"
│ time="2023-12-30T22:17:09Z" level=info msg="Removing container [rke-worker-port-listener] on host [192.168.1.41], try #1"
│ time="2023-12-30T22:17:09Z" level=info msg="[remove/rke-worker-port-listener] Successfully removed container on host [192.168.1.41]"
│ time="2023-12-30T22:17:09Z" level=info msg="[network] Port listener containers removed successfully"
│ time="2023-12-30T22:17:09Z" level=info msg="[certificates] Deploying kubernetes certificates to Cluster nodes"
│ time="2023-12-30T22:17:09Z" level=info msg="Finding container [cert-deployer] on host [192.168.1.41], try #1"
│ time="2023-12-30T22:17:09Z" level=info msg="Finding container [cert-deployer] on host [192.168.1.40], try #1"
│ time="2023-12-30T22:17:09Z" level=info msg="Image [registry.lab.reisman.org/proxy.docker.io/rancher/rke-tools:v0.1.96] exists on host [192.168.1.41]"
│ time="2023-12-30T22:17:09Z" level=info msg="Image [registry.lab.reisman.org/proxy.docker.io/rancher/rke-tools:v0.1.96] exists on host [192.168.1.40]"
│ time="2023-12-30T22:17:10Z" level=info msg="Starting container [cert-deployer] on host [192.168.1.41], try #1"
│ time="2023-12-30T22:17:10Z" level=info msg="Starting container [cert-deployer] on host [192.168.1.40], try #1"
│ time="2023-12-30T22:17:10Z" level=info msg="Finding container [cert-deployer] on host [192.168.1.41], try #1"
│ time="2023-12-30T22:17:11Z" level=info msg="Finding container [cert-deployer] on host [192.168.1.40], try #1"
│ time="2023-12-30T22:17:15Z" level=info msg="Finding container [cert-deployer] on host [192.168.1.41], try #1"
│ time="2023-12-30T22:17:15Z" level=info msg="Removing container [cert-deployer] on host [192.168.1.41], try #1"
│ time="2023-12-30T22:17:16Z" level=info msg="Finding container [cert-deployer] on host [192.168.1.40], try #1"
│ time="2023-12-30T22:17:16Z" level=info msg="Removing container [cert-deployer] on host [192.168.1.40], try #1"
│ time="2023-12-30T22:17:16Z" level=info msg="[reconcile] Rebuilding and updating local kube config"
│ time="2023-12-30T22:17:16Z" level=info msg="Successfully Deployed local admin kubeconfig at [/home/aaron/homelab-deployments/development/terraform-provider-rke-tmp-2841594254/kube_config_cluster.yml]"
│ time="2023-12-30T22:17:16Z" level=warning msg="[reconcile] host [192.168.1.40] is a control plane node without reachable Kubernetes API endpoint in the cluster"
│ time="2023-12-30T22:17:16Z" level=warning msg="[reconcile] no control plane node with reachable Kubernetes API endpoint in the cluster found"
│ time="2023-12-30T22:17:16Z" level=info msg="[certificates] Successfully deployed kubernetes certificates to Cluster nodes"
│ time="2023-12-30T22:17:16Z" level=info msg="[file-deploy] Deploying file [/etc/kubernetes/admission.yaml] to node [192.168.1.40]"
│ time="2023-12-30T22:17:16Z" level=info msg="Image [registry.lab.reisman.org/proxy.docker.io/rancher/rke-tools:v0.1.96] exists on host [192.168.1.40]"
│ time="2023-12-30T22:17:16Z" level=info msg="Starting container [file-deployer] on host [192.168.1.40], try #1"
│ time="2023-12-30T22:17:17Z" level=info msg="Successfully started [file-deployer] container on host [192.168.1.40]"
│ time="2023-12-30T22:17:17Z" level=info msg="Waiting for [file-deployer] container to exit on host [192.168.1.40]"
│ time="2023-12-30T22:17:17Z" level=info msg="Waiting for [file-deployer] container to exit on host [192.168.1.40]"
│ time="2023-12-30T22:17:17Z" level=info msg="Container [file-deployer] is still running on host [192.168.1.40]: stderr: [], stdout: []"
│ time="2023-12-30T22:17:18Z" level=info msg="Removing container [file-deployer] on host [192.168.1.40], try #1"
│ time="2023-12-30T22:17:18Z" level=info msg="[remove/file-deployer] Successfully removed container on host [192.168.1.40]"
│ time="2023-12-30T22:17:18Z" level=info msg="[/etc/kubernetes/admission.yaml] Successfully deployed admission control config to Cluster control nodes"
│ time="2023-12-30T22:17:18Z" level=info msg="[file-deploy] Deploying file [/etc/kubernetes/audit-policy.yaml] to node [192.168.1.40]"
│ time="2023-12-30T22:17:18Z" level=info msg="Image [registry.lab.reisman.org/proxy.docker.io/rancher/rke-tools:v0.1.96] exists on host [192.168.1.40]"
│ time="2023-12-30T22:17:19Z" level=info msg="Starting container [file-deployer] on host [192.168.1.40], try #1"
│ time="2023-12-30T22:17:19Z" level=info msg="Successfully started [file-deployer] container on host [192.168.1.40]"
│ time="2023-12-30T22:17:19Z" level=info msg="Waiting for [file-deployer] container to exit on host [192.168.1.40]"
│ time="2023-12-30T22:17:19Z" level=info msg="Waiting for [file-deployer] container to exit on host [192.168.1.40]"
│ time="2023-12-30T22:17:19Z" level=info msg="Container [file-deployer] is still running on host [192.168.1.40]: stderr: [], stdout: []"
│ time="2023-12-30T22:17:20Z" level=info msg="Removing container [file-deployer] on host [192.168.1.40], try #1"
│ time="2023-12-30T22:17:20Z" level=info msg="[remove/file-deployer] Successfully removed container on host [192.168.1.40]"
│ time="2023-12-30T22:17:20Z" level=info msg="[/etc/kubernetes/audit-policy.yaml] Successfully deployed audit policy file to Cluster control nodes"
│ time="2023-12-30T22:17:20Z" level=info msg="[reconcile] Reconciling cluster state"
│ time="2023-12-30T22:17:20Z" level=info msg="[reconcile] This is newly generated cluster"
│ time="2023-12-30T22:17:20Z" level=info msg="Pre-pulling kubernetes images"
│ time="2023-12-30T22:17:20Z" level=info msg="Pulling image [registry.lab.reisman.org/proxy.docker.io/rancher/hyperkube:v1.26.9-rancher1] on host [192.168.1.40], try #1"
│ time="2023-12-30T22:17:20Z" level=info msg="Pulling image [registry.lab.reisman.org/proxy.docker.io/rancher/hyperkube:v1.26.9-rancher1] on host [192.168.1.41], try #1"
│ time="2023-12-30T22:18:26Z" level=info msg="Image [registry.lab.reisman.org/proxy.docker.io/rancher/hyperkube:v1.26.9-rancher1] exists on host [192.168.1.40]"
│ time="2023-12-30T22:18:26Z" level=info msg="Pulling image [registry.lab.reisman.org/proxy.docker.io/rancher/mirrored-pause:3.7] on host [192.168.1.40], try #1"
│ time="2023-12-30T22:18:28Z" level=info msg="Image [registry.lab.reisman.org/proxy.docker.io/rancher/mirrored-pause:3.7] exists on host [192.168.1.40]"
│ time="2023-12-30T22:18:42Z" level=info msg="Image [registry.lab.reisman.org/proxy.docker.io/rancher/hyperkube:v1.26.9-rancher1] exists on host [192.168.1.41]"
│ time="2023-12-30T22:18:42Z" level=info msg="Pulling image [registry.lab.reisman.org/proxy.docker.io/rancher/mirrored-pause:3.7] on host [192.168.1.41], try #1"
│ time="2023-12-30T22:18:43Z" level=info msg="Image [registry.lab.reisman.org/proxy.docker.io/rancher/mirrored-pause:3.7] exists on host [192.168.1.41]"
│ time="2023-12-30T22:18:43Z" level=info msg="Kubernetes images pulled successfully"
│ time="2023-12-30T22:18:43Z" level=info msg="[etcd] Building up etcd plane.."
│ time="2023-12-30T22:18:43Z" level=info msg="Image [registry.lab.reisman.org/proxy.docker.io/rancher/rke-tools:v0.1.96] exists on host [192.168.1.40]"
│ time="2023-12-30T22:18:45Z" level=info msg="Starting container [etcd-fix-perm] on host [192.168.1.40], try #1"
│ time="2023-12-30T22:18:45Z" level=info msg="Successfully started [etcd-fix-perm] container on host [192.168.1.40]"
│ time="2023-12-30T22:18:45Z" level=info msg="Waiting for [etcd-fix-perm] container to exit on host [192.168.1.40]"
│ time="2023-12-30T22:18:45Z" level=info msg="Waiting for [etcd-fix-perm] container to exit on host [192.168.1.40]"
│ time="2023-12-30T22:18:46Z" level=info msg="Removing container [etcd-fix-perm] on host [192.168.1.40], try #1"
│ time="2023-12-30T22:18:46Z" level=info msg="[remove/etcd-fix-perm] Successfully removed container on host [192.168.1.40]"
│ time="2023-12-30T22:18:46Z" level=info msg="Pulling image [registry.lab.reisman.org/proxy.docker.io/rancher/mirrored-coreos-etcd:v3.5.6] on host [192.168.1.40], try #1"
│ time="2023-12-30T22:18:56Z" level=info msg="Image [registry.lab.reisman.org/proxy.docker.io/rancher/mirrored-coreos-etcd:v3.5.6] exists on host [192.168.1.40]"
│ time="2023-12-30T22:18:56Z" level=info msg="Starting container [etcd] on host [192.168.1.40], try #1"
│ time="2023-12-30T22:18:57Z" level=info msg="[etcd] Successfully started [etcd] container on host [192.168.1.40]"
│ time="2023-12-30T22:18:57Z" level=info msg="[etcd] Running rolling snapshot container [etcd-rolling-snapshots] on host [192.168.1.40]"
│ time="2023-12-30T22:18:57Z" level=info msg="Image [registry.lab.reisman.org/proxy.docker.io/rancher/rke-tools:v0.1.96] exists on host [192.168.1.40]"
│ time="2023-12-30T22:18:57Z" level=info msg="Starting container [etcd-rolling-snapshots] on host [192.168.1.40], try #1"
│ time="2023-12-30T22:18:58Z" level=info msg="[etcd] Successfully started [etcd-rolling-snapshots] container on host [192.168.1.40]"
│ time="2023-12-30T22:19:03Z" level=info msg="Image [registry.lab.reisman.org/proxy.docker.io/rancher/rke-tools:v0.1.96] exists on host [192.168.1.40]"
│ time="2023-12-30T22:19:04Z" level=info msg="Starting container [rke-bundle-cert] on host [192.168.1.40], try #1"
│ time="2023-12-30T22:19:04Z" level=info msg="[certificates] Successfully started [rke-bundle-cert] container on host [192.168.1.40]"
│ time="2023-12-30T22:19:04Z" level=info msg="Waiting for [rke-bundle-cert] container to exit on host [192.168.1.40]"
│ time="2023-12-30T22:19:04Z" level=info msg="[certificates] successfully saved certificate bundle [/opt/rke/etcd-snapshots//pki.bundle.tar.gz] on host [192.168.1.40]"
│ time="2023-12-30T22:19:04Z" level=info msg="Removing container [rke-bundle-cert] on host [192.168.1.40], try #1"
│ time="2023-12-30T22:19:04Z" level=info msg="Image [registry.lab.reisman.org/proxy.docker.io/rancher/rke-tools:v0.1.96] exists on host [192.168.1.40]"
│ time="2023-12-30T22:19:05Z" level=info msg="Starting container [rke-log-linker] on host [192.168.1.40], try #1"
│ time="2023-12-30T22:19:06Z" level=info msg="[etcd] Successfully started [rke-log-linker] container on host [192.168.1.40]"
│ time="2023-12-30T22:19:06Z" level=info msg="Removing container [rke-log-linker] on host [192.168.1.40], try #1"
│ time="2023-12-30T22:19:06Z" level=info msg="[remove/rke-log-linker] Successfully removed container on host [192.168.1.40]"
│ time="2023-12-30T22:19:06Z" level=info msg="Image [registry.lab.reisman.org/proxy.docker.io/rancher/rke-tools:v0.1.96] exists on host [192.168.1.40]"
│ time="2023-12-30T22:19:07Z" level=info msg="Starting container [rke-log-linker] on host [192.168.1.40], try #1"
│ time="2023-12-30T22:19:07Z" level=info msg="[etcd] Successfully started [rke-log-linker] container on host [192.168.1.40]"
│ time="2023-12-30T22:19:07Z" level=info msg="Removing container [rke-log-linker] on host [192.168.1.40], try #1"
│ time="2023-12-30T22:19:08Z" level=info msg="[remove/rke-log-linker] Successfully removed container on host [192.168.1.40]"
│ time="2023-12-30T22:19:08Z" level=info msg="[etcd] Successfully started etcd plane.. Checking etcd cluster health"
│ time="2023-12-30T22:19:08Z" level=info msg="[etcd] etcd host [192.168.1.40] reported healthy=true"
│ time="2023-12-30T22:19:08Z" level=info msg="[controlplane] Building up Controller Plane.."
│ time="2023-12-30T22:19:08Z" level=info msg="Finding container [service-sidekick] on host [192.168.1.40], try #1"
│ time="2023-12-30T22:19:08Z" level=info msg="Image [registry.lab.reisman.org/proxy.docker.io/rancher/rke-tools:v0.1.96] exists on host [192.168.1.40]"
│ time="2023-12-30T22:19:08Z" level=info msg="Image [registry.lab.reisman.org/proxy.docker.io/rancher/hyperkube:v1.26.9-rancher1] exists on host [192.168.1.40]"
│ time="2023-12-30T22:19:08Z" level=info msg="Starting container [kube-apiserver] on host [192.168.1.40], try #1"
│ time="2023-12-30T22:19:09Z" level=info msg="[controlplane] Successfully started [kube-apiserver] container on host [192.168.1.40]"
│ time="2023-12-30T22:19:09Z" level=info msg="[healthcheck] Start Healthcheck on service [kube-apiserver] on host [192.168.1.40]"
│ time="2023-12-30T22:19:19Z" level=info msg="[healthcheck] service [kube-apiserver] on host [192.168.1.40] is healthy"
│ time="2023-12-30T22:19:19Z" level=info msg="Image [registry.lab.reisman.org/proxy.docker.io/rancher/rke-tools:v0.1.96] exists on host [192.168.1.40]"
│ time="2023-12-30T22:19:20Z" level=info msg="Starting container [rke-log-linker] on host [192.168.1.40], try #1"
│ time="2023-12-30T22:19:21Z" level=info msg="[controlplane] Successfully started [rke-log-linker] container on host [192.168.1.40]"
│ time="2023-12-30T22:19:21Z" level=info msg="Removing container [rke-log-linker] on host [192.168.1.40], try #1"
│ time="2023-12-30T22:19:21Z" level=info msg="[remove/rke-log-linker] Successfully removed container on host [192.168.1.40]"
│ time="2023-12-30T22:19:21Z" level=info msg="Image [registry.lab.reisman.org/proxy.docker.io/rancher/hyperkube:v1.26.9-rancher1] exists on host [192.168.1.40]"
│ time="2023-12-30T22:19:21Z" level=info msg="Starting container [kube-controller-manager] on host [192.168.1.40], try #1"
│ time="2023-12-30T22:19:22Z" level=info msg="[controlplane] Successfully started [kube-controller-manager] container on host [192.168.1.40]"
│ time="2023-12-30T22:19:22Z" level=info msg="[healthcheck] Start Healthcheck on service [kube-controller-manager] on host [192.168.1.40]"
│ time="2023-12-30T22:19:27Z" level=info msg="[healthcheck] service [kube-controller-manager] on host [192.168.1.40] is healthy"
│ time="2023-12-30T22:19:27Z" level=info msg="Image [registry.lab.reisman.org/proxy.docker.io/rancher/rke-tools:v0.1.96] exists on host [192.168.1.40]"
│ time="2023-12-30T22:19:28Z" level=info msg="Starting container [rke-log-linker] on host [192.168.1.40], try #1"
│ time="2023-12-30T22:19:28Z" level=info msg="[controlplane] Successfully started [rke-log-linker] container on host [192.168.1.40]"
│ time="2023-12-30T22:19:28Z" level=info msg="Removing container [rke-log-linker] on host [192.168.1.40], try #1"
│ time="2023-12-30T22:19:28Z" level=info msg="[remove/rke-log-linker] Successfully removed container on host [192.168.1.40]"
│ time="2023-12-30T22:19:28Z" level=info msg="Image [registry.lab.reisman.org/proxy.docker.io/rancher/hyperkube:v1.26.9-rancher1] exists on host [192.168.1.40]"
│ time="2023-12-30T22:19:28Z" level=info msg="Starting container [kube-scheduler] on host [192.168.1.40], try #1"
│ time="2023-12-30T22:19:28Z" level=info msg="[controlplane] Successfully started [kube-scheduler] container on host [192.168.1.40]"
│ time="2023-12-30T22:19:28Z" level=info msg="[healthcheck] Start Healthcheck on service [kube-scheduler] on host [192.168.1.40]"
│ time="2023-12-30T22:19:34Z" level=info msg="[healthcheck] service [kube-scheduler] on host [192.168.1.40] is healthy"
│ time="2023-12-30T22:19:34Z" level=info msg="Image [registry.lab.reisman.org/proxy.docker.io/rancher/rke-tools:v0.1.96] exists on host [192.168.1.40]"
│ time="2023-12-30T22:19:35Z" level=info msg="Starting container [rke-log-linker] on host [192.168.1.40], try #1"
│ time="2023-12-30T22:19:35Z" level=info msg="[controlplane] Successfully started [rke-log-linker] container on host [192.168.1.40]"
│ time="2023-12-30T22:19:35Z" level=info msg="Removing container [rke-log-linker] on host [192.168.1.40], try #1"
│ time="2023-12-30T22:19:35Z" level=info msg="[remove/rke-log-linker] Successfully removed container on host [192.168.1.40]"
│ time="2023-12-30T22:19:35Z" level=info msg="[controlplane] Successfully started Controller Plane.."
│ time="2023-12-30T22:19:35Z" level=info msg="[authz] Creating rke-job-deployer ServiceAccount"
│ time="2023-12-30T22:19:35Z" level=info msg="[authz] rke-job-deployer ServiceAccount created successfully"
│ time="2023-12-30T22:19:35Z" level=info msg="[authz] Creating system:node ClusterRoleBinding"
│ time="2023-12-30T22:19:35Z" level=info msg="[authz] system:node ClusterRoleBinding created successfully"
│ time="2023-12-30T22:19:35Z" level=info msg="[authz] Creating kube-apiserver proxy ClusterRole and ClusterRoleBinding"
│ time="2023-12-30T22:19:35Z" level=info msg="[authz] kube-apiserver proxy ClusterRole and ClusterRoleBinding created successfully"
│ time="2023-12-30T22:19:35Z" level=info msg="Successfully Deployed state file at [/home/aaron/homelab-deployments/development/terraform-provider-rke-tmp-2841594254/cluster.rkestate]"
│ time="2023-12-30T22:19:35Z" level=info msg="[state] Saving full cluster state to Kubernetes"
│
│ Failed running cluster err:[state] Timeout waiting for kubernetes to be ready
│ ========================================
│
│
│ with module.k8s_cluster.rke_cluster.cluster,
│ on .terraform/modules/k8s_cluster/cluster.tf line 11, in resource "rke_cluster" "cluster":
│ 11: resource "rke_cluster" "cluster" {
I enabled debugging and retried the deploy. The logs still show that the last action is Saving full cluster state to Kubernetes, but now I'm getting the following error message.
│ Error:
│ ============= RKE outputs ==============
│ time="2024-01-03T04:36:01Z" level=debug msg="Loading data.json from local source"
│ time="2024-01-03T04:36:01Z" level=debug msg="data.json SHA256 checksum: 90b07a2e3abc57d43ca70fad7dfd980e889463b68af5027887af44363d457088"
│ time="2024-01-03T04:36:01Z" level=debug msg="Loading data.json from local source"
│ time="2024-01-03T04:36:01Z" level=debug msg="data.json SHA256 checksum: 90b07a2e3abc57d43ca70fad7dfd980e889463b68af5027887af44363d457088"
│ time="2024-01-03T04:36:02Z" level=debug msg="Loading data.json from local source"
│ time="2024-01-03T04:36:02Z" level=debug msg="data.json SHA256 checksum: 90b07a2e3abc57d43ca70fad7dfd980e889463b68af5027887af44363d457088"
│ time="2024-01-03T04:36:02Z" level=debug msg="Loading data.json from local source"
│ time="2024-01-03T04:36:02Z" level=debug msg="data.json SHA256 checksum: 90b07a2e3abc57d43ca70fad7dfd980e889463b68af5027887af44363d457088"
│ time="2024-01-03T04:36:02Z" level=info msg="[rke_provider] rke cluster changed arguments: map[addons:true ingress:true kubernetes_version:true nodes:true private_registries:true]"
│ time="2024-01-03T04:36:02Z" level=debug msg="[rke_provider] nodes values old: [] new: [map[address:192.168.1.40 docker_socket: hostname_override: internal_address:192.168.1.40 labels:map[] node_name: port: role:[controlplane etcd] roles: ssh_agent_auth:false ssh_cert: ssh_cert_path: ssh_key:-----BEGIN OPENSSH PRIVATE KEY-----[REDACTED]-----END OPENSSH PRIVATE KEY-----\n ssh_key_path: taints:[] user:aaron] map[address:192.168.1.41 docker_socket: hostname_override: internal_address:192.168.1.41 labels:map[] node_name: port: role:[worker] roles: ssh_agent_auth:false ssh_cert: ssh_cert_path: ssh_key:-----BEGIN OPENSSH PRIVATE KEY-----[REDACTED]-----END OPENSSH PRIVATE KEY-----\n ssh_key_path: taints:[] user:aaron]]"
│ time="2024-01-03T04:36:02Z" level=debug msg="[rke_provider] private_registries values old: [] new: [map[is_default:true password: url:registry.lab.reisman.org/proxy.docker.io user:]]"