Giters

xdavidel / k8s-vagrant

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

K8s Configuration

Table of contents

Instructions

All of the files in this repository are created by emacs using org-babel-tangle. It is recommaneded to change this README file and generate the changes into the files.

One can use the following to use org-babel-tangle from shell:

emacs -Q "README.org" --batch -f org-babel-tangle

Creating helper scripts

Common Scripts

Those scripts are used to initialize the containers for the cluster.

Pre run script

Some pakcages like grub we want to hold back from updating. After that we’ll update the kernel.

Update apt registry.

#!/bin/sh

apt-get update

Stop those packages from being updated

apt-mark hold package grub-pc grub-pc-bin grub2-common grub-common

Upgrade packages and kernel.

apt-get dist-upgrade -y

Install docker

Install packages to allow apt to use a repository over HTTPS.

#!/bin/sh

apt-get install -y apt-transport-https ca-certificates curl gnupg-agent software-properties-common

Add Docker apt repository.

curl -fsSL https://download.docker.com/linux/ubuntu/gpg | gpg --dearmor -o /usr/share/keyrings/docker-archive-keyring.gpg
echo "deb [arch=amd64 signed-by=/usr/share/keyrings/docker-archive-keyring.gpg] https://download.docker.com/linux/ubuntu $(lsb_release -cs) stable" | sudo tee /etc/apt/sources.list.d/docker.list > /dev/null
apt-get update

Install Docker CE.

apt-get install -y \
  containerd.io=1.4.6-1 \
  docker-ce=5:20.10.7~3-0~ubuntu-$(lsb_release -cs) \
  docker-ce-cli=5:20.10.7~3-0~ubuntu-$(lsb_release -cs)

Setup docker daemon

# Setup daemon.
cat > /etc/docker/daemon.json <<EOF
{
  "exec-opts": ["native.cgroupdriver=systemd"],
  "log-driver": "json-file",
  "log-opts": {
    "max-size": "100m"
  },
  "storage-driver": "overlay2"
}
EOF

mkdir -p /etc/systemd/system/docker.service.d

# Restart and enable docker service.
systemctl daemon-reload
systemctl start docker
systemctl enable docker

# Hold Docker at this specific version.
apt-mark hold docker-ce

Install kubernetes tools

Install packages to allow apt to use a repository over HTTPS

#!/bin/sh

apt-get install -y apt-transport-https curl

Add Kubernetes apt repository.

## Download the Google Cloud public signing key
curl -fsSLo /usr/share/keyrings/kubernetes-archive-keyring.gpg https://packages.cloud.google.com/apt/doc/apt-key.gpg

## Add the Kubernetes apt repository
echo "deb [signed-by=/usr/share/keyrings/kubernetes-archive-keyring.gpg] https://apt.kubernetes.io/ kubernetes-xenial main" | sudo tee /etc/apt/sources.list.d/kubernetes.list

## Update apt package index with the new repository
apt-get update

Install kubelet, kubeadm and kubectl.

apt-get install -y kubelet=1.21.2-00 kubeadm=1.21.2-00 kubectl=1.21.2-00

# Hold the Kubernetes components at this specific version.
apt-mark hold kubelet kubeadm kubectl

Turn off swap for kubeadm.

swapoff -a
sed -i '/swap/d' /etc/fstab

Post run script

#!/bin/bash

# Clear apt cache.
apt-get clean

# Cleanup disk.
#dd if=/dev/zero of=/EMPTY bs=1M
#rm -f /EMPTY

# Clear bash history.
cat /dev/null > ~/.bash_history && history -c && exit

Prepare kubectl

#!/bin/sh

# Prepare kubectl.
sudo mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config

Master scripts

Initialize master

#!/bin/sh

# Install kubernetes via kubeadm.
kubeadm init --apiserver-advertise-address=$NODE_IP

# Hostname -i must return a routable address on second (non-NATed) network interface.
# @see http://kubernetes.io/docs/getting-started-guides/kubeadm/#limitations
sed "s/127.0.0.1.*m/$NODE_IP m/" -i /etc/hosts

# Export k8s cluster token to an external file.
OUTPUT_FILE=/vagrant/join
rm -rf /vagrant/join
kubeadm token create --print-join-command > /vagrant/join
chmod +x $OUTPUT_FILE

Install CNI

#!/bin/sh

# Apply flannel.
kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml

Slave scripts

Initialize slave

#!/bin/sh

# Join kubernetes cluster.
/vagrant/join
echo "Environment='KUBELET_EXTRA_ARGS=--node-ip=$NODE_IP'" | tee -a /etc/systemd/system/kubelet.service.d/10-kubeadm.conf
systemctl daemon-reload
systemctl restart kubelet

Join kubernetes cluster

#!/bin/sh
kubeadm join 10.0.0.101:6443 --token 6oh2pt.iq2504a10odtxteq --discovery-token-ca-cert-hash sha256:5965c2c844a903e595bfd49e8aecbfed0cc7e57098c72044d63e308715d42896

Creating a Vagrantfile

Use the text editor of your choice and create a file with named Vagrantfile, inserting the code below. The value of N denotes the number of nodes present in the cluster, it can be modified accordingly. In the below example, we are setting the value of N as 2.

IMAGE_NAME="bento/ubuntu-18.04"
MEMORY_SIZE_IN_GB=2
CPU_COUNT=2
MASTER_NODE_COUNT=1
WORKER_NODE_COUNT=2
MASTER_NODE_IP_START="10.0.0.10"
WORKER_NODE_IP_START="10.0.0.20"

Vagrant.configure("2") do |config|

  # set variables
  master_node_ip = ''
  worker_node_ip = ''

  config.vm.box = IMAGE_NAME

  config.vm.provider "virtualbox" do |vb|

    vb.memory = 1024 * MEMORY_SIZE_IN_GB
    vb.cpus = CPU_COUNT

  end

  config.vm.provision "shell", path: "pre"

  config.vm.provision "shell", path: "install-docker"
  config.vm.provision "shell", path: "install-kube-tools"

  config.vm.provision "shell", path: "post"

  (1..MASTER_NODE_COUNT).each do |i|
    config.vm.define "m" do |master|

      master_node_ip = "#{MASTER_NODE_IP_START}#{i}"
      master.vm.network "private_network", ip: "#{master_node_ip}"
      master.vm.hostname = "m"

      # init master node.
      master.vm.provision "shell", path: "init-master-node", env: {"NODE_IP" => "#{master_node_ip}"}

      # prepare kubectl for vagrant user
      master.vm.provision "shell", privileged: false, path: "prepare-kubectl"

      # prepare kubectl for root user
      master.vm.provision "shell", privileged: true, path: "prepare-kubectl"

      # install cni.
      master.vm.provision "shell", path: "install-cni"

    end
  end

  (1..WORKER_NODE_COUNT).each do |i|
    config.vm.define "n#{i}" do |node|

      worker_node_ip = "#{WORKER_NODE_IP_START}#{i}"
      node.vm.network "private_network", ip: "#{worker_node_ip}"
      node.vm.hostname = "n#{i}"

      # init slave node.
      node.vm.provision "shell", path: "init-slave-node", env: {"NODE_IP" => "#{worker_node_ip}"}

    end
  end
end

Checking the resulted cluster

Deploying vagrant configuration

cd /path/to/Vagrantfile
vagrant up m n1 n2

ssh into the master node

Upon completion of all the above steps, the Kubernetes cluster should be up and running. We can login to the master or worker nodes using Vagrant as follows:

## Accessing master
vagrant ssh m

We can check all the nodes through the master:

vagrant@k8s-master:~$ kubectl get nodes -o wide
NAME   STATUS     ROLES                  AGE     VERSION
m      Ready      control-plane,master   2d18h   v1.21.2
n1     NotReady   <none>                 2d18h   v1.21.2
n2     NotReady   <none>                 2d18h   v1.21.2


vagrant@m:~$ kubectl get nodes -o wide
NAME   STATUS     ROLES                  AGE     VERSION   INTERNAL-IP   EXTERNAL-IP   OS-IMAGE             KERNEL-VERSION       CONTAINER-RUNTIME
m      Ready      control-plane,master   2d18h   v1.21.2   10.0.2.15     <none>        Ubuntu 18.04.6 LTS   4.15.0-163-generic   docker://20.10.7
n1     NotReady   <none>                 2d18h   v1.21.2   10.0.0.201    <none>        Ubuntu 18.04.6 LTS   4.15.0-163-generic   docker://20.10.7
n2     NotReady   <none>                 2d18h   v1.21.2   10.0.0.202    <none>        Ubuntu 18.04.6 LTS   4.15.0-163-generic   docker://20.10.7

ssh into ant other node

## Accessing nodes
vagrant ssh n1
vagrant ssh n2

About

Languages

Language:Shell 100.0%