Giters

MichaelCade / argocd-kanister

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

Integrating Backup into your CI/CD Pipelines using Kanister

Deploy local Minikube cluster

minikube start --addons volumesnapshots,csi-hostpath-driver --apiserver-port=6443 --container-runtime=containerd -p mc-demo --kubernetes-version=1.21.2

How to Install Kanister

add kanister helm repository

helm repo add kanister https://charts.kanister.io/

create kanister namespace

kubectl create namespace kanister

deploy kanister using helm

helm install myrelease --namespace kanister kanister/kanister-operator --set image.tag=0.71.0

Once we have kanister deployed we should now show the CustomResourceDefinitions, this will show actionsets, blueprints, profiles.

kubectl get customresourcedefinitions.apiextensions.k8s.io | grep "kanister"

Deploy ArgoCD

kubectl create namespace argocd
kubectl apply -n argocd -f https://raw.githubusercontent.com/argoproj/argo-cd/stable/manifests/install.yaml
kubectl port-forward svc/argocd-server -n argocd 8443:443

Username is admin and password can be obtained with this command. open a web browser localhost

kubectl -n argocd get secret argocd-initial-admin-secret -o jsonpath="{.data.password}" | base64 -d && echo

Create a bucket in minikube with minio

Enable load balancer in minikube

# run this command in another terminal
minikube tunnel --profile=mc-demo

Install minio

helm repo add minio https://charts.min.io/
kubectl create ns minio
helm install kasten-minio minio/minio --namespace=minio --version 8.0.10 \
  --set persistence.size=5Gi

# feed ACCESS_KEY variable and SECRET_KEY and display them
AWS_ACCESS_KEY_ID=$(kubectl -n minio get secret kasten-minio -o jsonpath="{.data.accesskey}" | base64 --decode)
echo $AWS_ACCESS_KEY_ID
AWS_SECRET_KEY=$(kubectl -n minio get secret kasten-minio -o jsonpath="{.data.secretkey}" | base64 --decode)
echo $AWS_SECRET_KEY

# Expose minio in a load balancer 
kubectl expose svc/kasten-minio --name=minio-lb -n minio --type=LoadBalancer
EXTERNAL_IP=$(kubectl get svc -n minio minio-lb -o jsonpath='{.spec.clusterIP}')
echo "http://$EXTERNAL_IP:9000/"

open a web browser to this addres and use $ACCESS_KEYand $SECRET_KEY to access th minio GUI.

Create a kanister bucket.

Create a Kanister Profile

This will give us some where to store our backups, this will be done using the KanCTL CLI tool

kanctl create profile \
   --namespace kanister \
   --bucket kanister \
   --skip-SSL-verification \
   --endpoint http://$EXTERNAL_IP:9000/ \
   s3compliant \
   --access-key $AWS_ACCESS_KEY_ID \
   --secret-key $AWS_SECRET_KEY

confirm you now have a profile created

kubectl get profile -n kanister

Creating a Kanister Blueprint

Kanister uses Blueprints to define these database-specific workflows and open-source Blueprints are available for several popular applications. It's also simple to customize existing Blueprints or add new ones.

kubectl create -f https://raw.githubusercontent.com/kanisterio/kanister/master/examples/stable/mysql/mysql-blueprint.yaml -n kanister

this blueprint is created in the Kanister namespace check with environment

kubectl get blueprint -n kanister

Use argo to deploy your app

Create the argo project

Create a mysql namespace.

kubectl create ns mysql

Create in argo a new project mysql-app :

Once deployed, check the service account kanister-presync can create an action or read a profile set in the kanister namespace.

kubectl auth can-i create actionset --as=system:serviceaccount:mysql:kanister-presync -n kanister
kubectl auth can-i create profile --as=system:serviceaccount:mysql:kanister-presync -n kanister

The answer should be yes for both.

Create some data

kubectl exec -ti mysql-0 -n mysql -- bash

mysql --user=root --password=ultrasecurepassword
CREATE DATABASE test;
USE test;
CREATE TABLE pets (name VARCHAR(20), owner VARCHAR(20), species VARCHAR(20), sex CHAR(1), birth DATE, death DATE);
INSERT INTO pets VALUES ('Puffball','Diane','hamster','f','1999-03-30',NULL);
SELECT * FROM pets;
+----------+-------+---------+------+------------+-------+
| name     | owner | species | sex  | birth      | death |
+----------+-------+---------+------+------------+-------+
| Puffball | Diane | hamster | f    | 1999-03-30 | NULL  |
+----------+-------+---------+------+------------+-------+
1 row in set (0.00 sec)
exit
exit

Sync your project

By resyncing your project you're going to trigger the creation of a backup. check on minio.

Introduce some "bad" change in your application stack.

Let's imagine you create a mysqlclient app which is going to drop your database in your code, that's a mistake. But mistake happen.

Create this pod base/mysql-client.yaml

apiVersion: v1
kind: Pod
metadata:
  labels:
    run: mysql-client
  name: mysql-client
spec:
  containers:
  - image: mysql:8.0.26
    name: mysql-client
    env:
    - name: MYSQL_ROOT_PASSWORD
      valueFrom:
        secretKeyRef:
          key: mysql-root-password
          name: mysql
    command: 
      - sh
      - -o
      - errexit
      - -c
      - |         
        mysql -h mysql --user=root --password=$MYSQL_ROOT_PASSWORD -e "DROP DATABASE test;"
  dnsPolicy: ClusterFirst
  restartPolicy: Never
status: {}

You commit, push and sync with argo and check your data

kubectl exec -ti mysql-0 -n mysql -- bash

mysql --user=root --password=ultrasecurepassword
SHOW DATABASES;

+--------------------+
| Database           |
+--------------------+
| information_schema |
| mysql              |
| performance_schema |
| sys                |
+--------------------+
4 rows in set (0.00 sec)

Horror !!! The sync has deleted the database but fortunaletly kanister protects your database in the execution of the presync.

Restore your database using kanctl

The sync above represents a simple change in code that could affect our data, at this stage the bad mysql-client.yaml should be removed or configured correctly before continuing with the restore process.

Before we restore we need to choose the appropriate restore point.

kubectl get actionset -n kanister

NAME                                           AGE
mysqlapp-backup-tue-jan-25-17-46-46-utc-2022   10m
mysqlapp-backup-tue-jan-25-17-50-45-utc-2022   6m56s

When we have the list we can choose the correct actionset to restore from.

kanctl --namespace kanister create actionset --action restore --from "mysqlapp-backup-tue-jan-25-17-50-45-utc-2022"

actionset restore-mysqlapp-backup-tue-jan-25-17-50-45-utc-2022-hsjvv created

Now we can see in detail what is happening as part of the restore process.

kubectl --namespace kanister describe actionset restore-mysqlapp-backup-tue-jan-25-17-50-45-utc-2022-hsjvv

We can then go and check on our data and database again and make sure that our test database is restored along with our data.

kubectl exec -ti mysql-0 -n mysql -- bash

mysql --user=root --password=ultrasecurepassword
CREATE DATABASE test;
USE test;
SELECT * FROM pets;
+----------+-------+---------+------+------------+-------+
| name     | owner | species | sex  | birth      | death |
+----------+-------+---------+------+------------+-------+
| Puffball | Diane | hamster | f    | 1999-03-30 | NULL  |
+----------+-------+---------+------+------------+-------+
1 row in set (0.00 sec)

SHOW DATABASES;

exit
exit

About

Languages

Language:Shell 100.0%