Failed deployment of site.yml
stevenmcastano opened this issue · comments
While trying to run in site.yml playbook, it appears there is an issue installing metallb:
Jun 30 21:54:19 debian ansible-ansible.legacy.command: Invoked with _raw_params=k3s kubectl get nodes -l "node-role.kubernetes.io/master=true" -o=jsonpath="{.items[*].metadata.name}" _uses_shell=False stdin_add_newline=True strip_empty_ends=True argv=None chdir=None executable=None creates=None removes=None stdin=None
Jun 30 21:54:23 debian k3s[6502]: E0630 21:54:23.166542 6502 kubelet.go:1731] "Unable to attach or mount volumes for pod; skipping pod" err="unmounted volumes=[memberlist], unattached volumes=[memberlist kube-api-access-h5bc5]: timed out waiting for the condition" pod="metallb-system/speaker-55hvs"
Jun 30 21:54:23 debian k3s[6502]: E0630 21:54:23.166587 6502 pod_workers.go:965] "Error syncing pod, skipping" err="unmounted volumes=[memberlist], unattached volumes=[memberlist kube-api-access-h5bc5]: timed out waiting for the condition" pod="metallb-system/speaker-55hvs" podUID=d9bfce8a-9438-4fc0-a4e2-85c045c2bb30
Jun 30 21:54:30 debian ansible-ansible.legacy.command: Invoked with _raw_params=k3s kubectl get nodes -l "node-role.kubernetes.io/master=true" -o=jsonpath="{.items[*].metadata.name}" _uses_shell=False stdin_add_newline=True strip_empty_ends=True argv=None chdir=None executable=None creates=None removes=None stdin=None
Jun 30 21:54:40 debian ansible-ansible.legacy.command: Invoked with _raw_params=k3s kubectl get nodes -l "node-role.kubernetes.io/master=true" -o=jsonpath="{.items[*].metadata.name}" _uses_shell=False stdin_add_newline=True strip_empty_ends=True argv=None chdir=None executable=None creates=None removes=None stdin=None
Jun 30 21:54:51 debian ansible-ansible.legacy.command: Invoked with _raw_params=k3s kubectl get nodes -l "node-role.kubernetes.io/master=true" -o=jsonpath="{.items[*].metadata.name}" _uses_shell=False stdin_add_newline=True strip_empty_ends=True argv=None chdir=None executable=None creates=None removes=None stdin=None
Jun 30 21:55:01 debian ansible-ansible.legacy.command: Invoked with _raw_params=k3s kubectl get nodes -l "node-role.kubernetes.io/master=true" -o=jsonpath="{.items[*].metadata.name}" _uses_shell=False stdin_add_newline=True strip_empty_ends=True argv=None chdir=None executable=None creates=None removes=None stdin=None
Jun 30 21:55:12 debian ansible-ansible.legacy.command: Invoked with _raw_params=k3s kubectl get nodes -l "node-role.kubernetes.io/master=true" -o=jsonpath="{.items[*].metadata.name}" _uses_shell=False stdin_add_newline=True strip_empty_ends=True argv=None chdir=None executable=None creates=None removes=None stdin=None
Jun 30 21:55:22 debian ansible-ansible.legacy.command: Invoked with _raw_params=k3s kubectl get nodes -l "node-role.kubernetes.io/master=true" -o=jsonpath="{.items[*].metadata.name}" _uses_shell=False stdin_add_newline=True strip_empty_ends=True argv=None chdir=None executable=None creates=None removes=None stdin=None
Jun 30 21:55:24 debian k3s[6502]: E0630 21:55:24.092067 6502 event_broadcaster.go:253] Server rejected event '&v1.Event{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"controller-844979dcdc-r5j9j.176d9b54da9fb5dd", GenerateName:"", Namespace:"metallb-system", SelfLink:"", UID:"b0b134b2-c2f9-4619-80a2-166c4c9dc59e", ResourceVersion:"2245", Generation:0, CreationTimestamp:time.Date(2023, time.June, 30, 21, 50, 24, 0, time.Local), DeletionTimestamp:<nil>, DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string(nil), Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Finalizers:[]string(nil), ManagedFields:[]v1.ManagedFieldsEntry{v1.ManagedFieldsEntry{Manager:"k3s", Operation:"Update", APIVersion:"events.k8s.io/v1", Time:time.Date(2023, time.June, 30, 21, 50, 24, 0, time.Local), FieldsType:"FieldsV1", FieldsV1:(*v1.FieldsV1)(0xc0132e91e8), Subresource:""}}}, EventTime:time.Date(2023, time.June, 30, 21, 50, 24, 81259000, time.Local), Series:(*v1.EventSeries)(0xc00e20c400), ReportingController:"default-scheduler", ReportingInstance:"default-scheduler-debian", Action:"Scheduling", Reason:"FailedScheduling", Regarding:v1.ObjectReference{Kind:"Pod", Namespace:"metallb-system", Name:"controller-844979dcdc-r5j9j", UID:"628da9cb-3923-4e30-b7fa-4219f47ffb88", APIVersion:"v1", ResourceVersion:"532", FieldPath:""}, Related:(*v1.ObjectReference)(nil), Note:"0/1 nodes are available: 1 node(s) had untolerated taint {node-role.kubernetes.io/master: true}. preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling.", Type:"Warning", DeprecatedSource:v1.EventSource{Component:"", Host:""}, DeprecatedFirstTimestamp:time.Date(1, time.January, 1, 0, 0, 0, 0, time.UTC), DeprecatedLastTimestamp:time.Date(1, time.January, 1, 0, 0, 0, 0, time.UTC), DeprecatedCount:0}': 'Event "controller-844979dcdc-r5j9j.176d9b54da9fb5dd" is invalid: series.count: Invalid value: "": should be at least 2' (will not retry!)
Jun 30 21:55:33 debian ansible-ansible.legacy.command: Invoked with _raw_params=k3s kubectl get nodes -l "node-role.kubernetes.io/master=true" -o=jsonpath="{.items[*].metadata.name}" _uses_shell=False stdin_add_newline=True strip_empty_ends=True argv=None chdir=None executable=None creates=None removes=None stdin=None
Jun 30 21:55:44 debian ansible-ansible.legacy.command: Invoked with _raw_params=k3s kubectl get nodes -l "node-role.kubernetes.io/master=true" -o=jsonpath="{.items[*].metadata.name}" _uses_shell=False stdin_add_newline=True strip_empty_ends=True argv=None chdir=None executable=None creates=None removes=None stdin=None
Jun 30 21:55:54 debian ansible-ansible.legacy.command: Invoked with _raw_params=k3s kubectl get nodes -l "node-role.kubernetes.io/master=true" -o=jsonpath="{.items[*].metadata.name}" _uses_shell=False stdin_add_newline=True strip_empty_ends=True argv=None chdir=None executable=None creates=None removes=None stdin=None
Jun 30 21:56:05 debian ansible-ansible.legacy.command: Invoked with _raw_params=k3s kubectl get nodes -l "node-role.kubernetes.io/master=true" -o=jsonpath="{.items[*].metadata.name}" _uses_shell=False stdin_add_newline=True strip_empty_ends=True argv=None chdir=None executable=None creates=None removes=None stdin=None
Jun 30 21:56:15 debian ansible-ansible.legacy.command: Invoked with _raw_params=k3s kubectl get nodes -l "node-role.kubernetes.io/master=true" -o=jsonpath="{.items[*].metadata.name}" _uses_shell=False stdin_add_newline=True strip_empty_ends=True argv=None chdir=None executable=None creates=None removes=None stdin=None
Jun 30 21:56:17 debian k3s[6502]: E0630 21:56:17.252166 6502 secret.go:192] Couldn't get secret metallb-system/memberlist: secret "memberlist" not found
Jun 30 21:56:17 debian k3s[6502]: E0630 21:56:17.252286 6502 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d9bfce8a-9438-4fc0-a4e2-85c045c2bb30-memberlist podName:d9bfce8a-9438-4fc0-a4e2-85c045c2bb30 nodeName:}" failed. No retries permitted until 2023-06-30 21:58:19.252248896 -0400 EDT m=+515.916813259 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "memberlist" (UniqueName: "kubernetes.io/secret/d9bfce8a-9438-4fc0-a4e2-85c045c2bb30-memberlist") pod "speaker-55hvs" (UID: "d9bfce8a-9438-4fc0-a4e2-85c045c2bb30") : secret "memberlist" not found
Jun 30 21:56:26 debian ansible-ansible.legacy.command: Invoked with _raw_params=k3s kubectl get nodes -l "node-role.kubernetes.io/master=true" -o=jsonpath="{.items[*].metadata.name}" _uses_shell=False stdin_add_newline=True strip_empty_ends=True argv=None chdir=None executable=None creates=None removes=None stdin=None
Jun 30 21:56:36 debian ansible-ansible.legacy.command: Invoked with _raw_params=k3s kubectl get nodes -l "node-role.kubernetes.io/master=true" -o=jsonpath="{.items[*].metadata.name}" _uses_shell=False stdin_add_newline=True strip_empty_ends=True argv=None chdir=None executable=None creates=None removes=None stdin=None
Jun 30 21:56:37 debian k3s[6502]: E0630 21:56:37.166844 6502 kubelet.go:1731] "Unable to attach or mount volumes for pod; skipping pod" err="unmounted volumes=[memberlist], unattached volumes=[memberlist kube-api-access-h5bc5]: timed out waiting for the condition" pod="metallb-system/speaker-55hvs"
Jun 30 21:56:37 debian k3s[6502]: E0630 21:56:37.166893 6502 pod_workers.go:965] "Error syncing pod, skipping" err="unmounted volumes=[memberlist], unattached volumes=[memberlist kube-api-access-h5bc5]: timed out waiting for the condition" pod="metallb-system/speaker-55hvs" podUID=d9bfce8a-9438-4fc0-a4e2-85c045c2bb30
Jun 30 21:56:47 debian ansible-ansible.legacy.command: Invoked with _raw_params=k3s kubectl get nodes -l "node-role.kubernetes.io/master=true" -o=jsonpath="{.items[*].metadata.name}" _uses_shell=False stdin_add_newline=True strip_empty_ends=True argv=None chdir=None executable=None creates=None removes=None stdin=None
Jun 30 21:56:57 debian ansible-ansible.legacy.command: Invoked with _raw_params=k3s kubectl get nodes -l "node-role.kubernetes.io/master=true" -o=jsonpath="{.items[*].metadata.name}" _uses_shell=False stdin_add_newline=True strip_empty_ends=True argv=None chdir=None executable=None creates=None removes=None stdin=None
Jun 30 21:57:08 debian ansible-ansible.legacy.command: Invoked with _raw_params=k3s kubectl get nodes -l "node-role.kubernetes.io/master=true" -o=jsonpath="{.items[*].metadata.name}" _uses_shell=False stdin_add_newline=True strip_empty_ends=True argv=None chdir=None executable=None creates=None removes=None stdin=None
Jun 30 21:57:19 debian ansible-ansible.legacy.command: Invoked with _raw_params=k3s kubectl get nodes -l "node-role.kubernetes.io/master=true" -o=jsonpath="{.items[*].metadata.name}" _uses_shell=False stdin_add_newline=True strip_empty_ends=True argv=None chdir=None executable=None creates=None removes=None stdin=None
Jun 30 21:57:29 debian ansible-ansible.legacy.command: Invoked with _raw_params=k3s kubectl get nodes -l "node-role.kubernetes.io/master=true" -o=jsonpath="{.items[*].metadata.name}" _uses_shell=False stdin_add_newline=True strip_empty_ends=True argv=None chdir=None executable=None creates=None removes=None stdin=None
Jun 30 21:57:40 debian ansible-ansible.legacy.command: Invoked with _raw_params=k3s kubectl get nodes -l "node-role.kubernetes.io/master=true" -o=jsonpath="{.items[*].metadata.name}" _uses_shell=False stdin_add_newline=True strip_empty_ends=True argv=None chdir=None executable=None creates=None removes=None stdin=None
Jun 30 21:57:50 debian ansible-ansible.legacy.command: Invoked with _raw_params=k3s kubectl get nodes -l "node-role.kubernetes.io/master=true" -o=jsonpath="{.items[*].metadata.name}" _uses_shell=False stdin_add_newline=True strip_empty_ends=True argv=None chdir=None executable=None creates=None removes=None stdin=None
Jun 30 21:58:01 debian ansible-ansible.legacy.command: Invoked with _raw_params=k3s kubectl get nodes -l "node-role.kubernetes.io/master=true" -o=jsonpath="{.items[*].metadata.name}" _uses_shell=False stdin_add_newline=True strip_empty_ends=True argv=None chdir=None executable=None creates=None removes=None stdin=None
Jun 30 21:58:11 debian ansible-ansible.legacy.command: Invoked with _raw_params=k3s kubectl get nodes -l "node-role.kubernetes.io/master=true" -o=jsonpath="{.items[*].metadata.name}" _uses_shell=False stdin_add_newline=True strip_empty_ends=True argv=None chdir=None executable=None creates=None removes=None stdin=None
Jun 30 21:58:19 debian k3s[6502]: E0630 21:58:19.327428 6502 secret.go:192] Couldn't get secret metallb-system/memberlist: secret "memberlist" not found
Jun 30 21:58:19 debian k3s[6502]: E0630 21:58:19.327536 6502 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d9bfce8a-9438-4fc0-a4e2-85c045c2bb30-memberlist podName:d9bfce8a-9438-4fc0-a4e2-85c045c2bb30 nodeName:}" failed. No retries permitted until 2023-06-30 22:00:21.327513821 -0400 EDT m=+637.992078174 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "memberlist" (UniqueName: "kubernetes.io/secret/d9bfce8a-9438-4fc0-a4e2-85c045c2bb30-memberlist") pod "speaker-55hvs" (UID: "d9bfce8a-9438-4fc0-a4e2-85c045c2bb30") : secret "memberlist" not found
Jun 30 21:58:22 debian ansible-ansible.legacy.command: Invoked with _raw_params=k3s kubectl get nodes -l "node-role.kubernetes.io/master=true" -o=jsonpath="{.items[*].metadata.name}" _uses_shell=False stdin_add_newline=True strip_empty_ends=True argv=None chdir=None executable=None creates=None removes=None stdin=None
Jun 30 21:58:32 debian ansible-ansible.legacy.command: Invoked with _raw_params=k3s kubectl get nodes -l "node-role.kubernetes.io/master=true" -o=jsonpath="{.items[*].metadata.name}" _uses_shell=False stdin_add_newline=True strip_empty_ends=True argv=None chdir=None executable=None creates=None removes=None stdin=None
Jun 30 21:58:33 debian ansible-systemd: Invoked with name=k3s-init state=stopped daemon_reload=False daemon_reexec=False scope=system no_block=False enabled=None force=None masked=None
Jun 30 21:58:33 debian k3s[6502]: I0630 21:58:33.479965 6502 health_controller.go:193] Shutting down HealthController RunCheck
Jun 30 21:58:33 debian systemd[1]: Stopping /usr/local/bin/k3s server --cluster-init --token hv@YRHzSK#ukRU&jk!KG65MoUu@d&dfnTjD4MT9wpfq8tgjN7Lvd@JuaFrkB6E^krtVVuFBMW#@ --flannel-iface=ens18 --node-ip=192.168.1.191 --node-taint node-role.kubernetes.io/master=true:NoSchedule --tls-san 192.168.1.190 --disable servicelb --disable traefik...
Jun 30 21:58:33 debian systemd[1]: k3s-init.service: Succeeded.
Jun 30 21:58:33 debian systemd[1]: Stopped /usr/local/bin/k3s server --cluster-init --token hv@YRHzSK#ukRU&jk!KG65MoUu@d&dfnTjD4MT9wpfq8tgjN7Lvd@JuaFrkB6E^krtVVuFBMW#@ --flannel-iface=ens18 --node-ip=192.168.1.191 --node-taint node-role.kubernetes.io/master=true:NoSchedule --tls-san 192.168.1.190 --disable servicelb --disable traefik.
Jun 30 21:58:33 debian systemd[1]: k3s-init.service: Consumed 1min 18.516s CPU time.
Operating system:
Debian 11
Hardware:
Proxmox VM, 4 cores, 4GB ram
I have a similar looking issue with the error "metallb-system/memberlist: secret "memberlist" not found". I've tried spinning up both Ubuntu and Debian VMs but have the same issue on both. I kept the config just like the sample and only changed the IPs and secret. I also made sure all my VMs have eth0 as the interface. The VIP responds to pings but the setup isn't complete, and no kubectl.
Solution: I setup VMs using this guide. I believe the problem was that I cloned VMs to create the other nodes, so all the machines had the same identifiers.