cgraaaj / k8s-setup

Kubernetes-Home-Lab

A repository for the Ansible playbooks used in my Youtube video series. In this series I go over configuring my Raspberry Pi 4 cluster as a Kubernetes cluster using Kubeadm. I also automate the process as much as I can using these Ansible playbooks.

Each playbook role comes with a README file outling the manual commands as well if Ansible is not somethign the user wishes to use for this proces .

Link to the youtube series:

• Part 1: Design, Hardware, and Installing the OS
• Part 2: Configuring Bare Metal Hosts
• Part 3: Configuring Pre-requisites to run Kubeadm
• Part 3a: My Thought Process When Building Palybooks
• Part 4: Deploying an NGINX Load Balancer with Ansible
• Part 5: Creating Encryption Secrets for Kubernetes Cluster with Ansible
• Part 6: Creating a CA and certificates for Kubernetes Cluster with Ansible
• Part 7: Deploying Kubeadm and Joining Nodes
• Part 8: Creating an NFS server for persistent Storage

Architecure and Versions

• Ubuntu 20.04.2 LTS (GNU/Linux 5.4.0-1029-raspi aarch64)
• Kubeadm, kubectl, kubelet v1.21.0
• containerd .io v1.4.4
• Flannel ( k8s networking solution)
• Openssl v1.1.1f

Networking

-Cluster CIDR: IPs for pods - 10.240.0.0/16 -Worker Nodes Pod CIDR: Sepcific CIDR within the Cluster CIDR for one worker node. Multiple worker nodes pod CIDRs cannot overlap ( controlled by Flannel )

Known Issues

Issue:

• Kubernetes-dashboard pods stuck in creating

Error:

• Warning FailedCreatePodSandBox 13s (x6 over 79s) kubelet (combined from similar events): Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "1cce51808f870550bc01b3a8899e1fa13cad6372fa465a8e4c459a18ec8c51ea": failed to set bridge addr: "cni0" already has an IP address different from 10.240.1.1/24

Fix:

• This happens when you destroy flannel and deploy it again sometimes. Go to the nodes the kubernetes-dashboard hosts are hosted on and run sudo ip link set cni0 down and then sudo brctl delbr cni0 and wait a few minutes.

Issue:

• Flannel containers CrashLoppBackoff

Error:

• Error registering network: failed to configure interface flannel.1: failed to ensure address of interface flannel.1: link has incompatible addresses. Remove additional addresses and try again.

Fix:

• This happens when you destroy flannel and deploy it again sometimes. On the nodes in question run sudo ip link delete flannel.1 and then wait a few minutes for flannel pods to restart.

Issue:

• kubectl not working with error:

Error:

• The connection to the server localhost:8080 was refused - did you specify the right host or port?

Fix:

• This means the admin.conf for your cluster doesnt exist in that users $HOME/.kube directory. If Kubeadm did NOT finish the init this wont be available yet. If it did finish the init, simply place it in the directory cp /etc/kubernetes/admin.conf $HOME/.kube/

Issue:

• kubelet cannot connect to API server address:6443

Error:

• This shows up under journalctl -u kubelet or systemctl status kubelet as connection refused to the kube-api-server address on port 6443.

Fix:

• This one can happen for multiple reasons, but the best way to get the bottom of it is to first verify the kube-api container is healthy crictl ps -a will show you the running containers on the mastern node. If the kube-api one is in a state other than "running" you can do a crictl logs < kube-api container ID > to get more info about why the container is not creating. It may be due to a bad option or syntax passed into the config used in kubeadm init --config.
• It may also be due to firewall issues. Check to make sure port 6443 is accesible.

Issue:

• Nodes not ready when using kubectl get nodes

Error:

• Container runtime network not ready" networkReady="NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: cni plugin not initialized

Fix:

• Did you install networking such as Netweave, flannel, calico, etc? If not this is needed.
• Did you delete /opt/cni/ and now its empty? You should see a "bin" directory. If you did delete it, then reinstall kubernetes-cni.
• Try and restart containerd as well then check for the cni0 interface.

Troubleshooting

Get logs from a container using crictl

• crictl ps -a to list running containers and their ID
• crictl logs < container ID > to view logs for the container
• kubectl get nodes will list the nodes and their READY or NOT READY status in the cluster
• kubectl describe node < node > will show more information about the node and possible reveal any issues
• kubectl get pods -n kube-system -o wide A widely used command to display all pods in the kube-system namespace and their status

Continue here for more https://github.com/justmeandopensource/kubernetes