Codefresh GitOps Certification examples - Level 2 - GitOps at scale

This repository contains examples for the ArgoCD/GitOps certification workshops (Level 2)

Take the certification yourself at https://codefresh.io/courses/get-gitops-certified/

2. Handling multiple Applications

App of Apps

argocd app create my-favorite-apps \
--repo https://github.com/codefresh-contrib/gitops-cert-level-2-examples \
--project default \
--sync-policy none \
--path ./app-of-apps/my-app-list \
--dest-namespace argocd \
--dest-server https://kubernetes.default.svc

Parent app my-favourite-apps has a manual sync policy, the child apps have automatic sync policies
Changes in the child app manifests will trigger an auto syncornization, changes to the child app definitions will NOT becuase they are synced using our parent app.

Mutli Cluster Management

After initial installation Argo CD can automatically deploy to its own cluster (referred to as “internal”)
You can add additional external clusters (public/local) with the Argo CD CLI, you first need to ensure you have a valid context in your kubeconfig for the cluster.

# Creates new Service account on the target cluster and add that target cluster to ArgoCD
argocd cluster add <context name>

# List clusters
argocd cluster list

# Remove a cluster
# Removing a cluster does not remove the Argo CD Applications associated with it.
argocd cluster rm <server>

Managing grouped Argo CD instances with multiple clusters

Argo CD Projects provide a way to group Argo CD applications when organizations have multiple teams using Argo
Projects allow you to restrict what is deployed, where apps can be deployed, and what sort of objects (CRDs, RBAC, etc.) can be deployed.

# Adding a second cluster
# Login to the first cluster from the 2nd cluster
# kubernetes-vm:30443 represents main cluster
argocd login kubernetes-vm:30443 --insecure --username admin --password <passwd>

# Add the cluster - this results in argocd-manager service account created on cluster-2 in kube-system namespace
argocd cluster add default --name cluster-2

# Adding new app to cluster 2
# Executed from main cluster
argocd app create external-app \
--repo https://github.com/theJaxon/gitops-cert-level-2-examples \
--project default \
--sync-policy automatic \
--path ./simple-application \
--dest-namespace default \
--dest-server https://10.5.1.144:6443 # cluster-2 external IP

# Adding new app to our main cluster
argocd app create internal-app \
--repo https://github.com/theJaxon/gitops-cert-level-2-examples \
--project default \
--sync-policy automatic \
--path ./simple-application \
--dest-namespace default \
--dest-server https://kubernetes.default.svc

ApplicationSet Generators

An ApplicationSet uses templated automation to create, modify, and manage multiple Argo CD applications at once, while also targeting multiple clusters and namespaces.
It's made up of generators that generate the application.
Generators are responsible for providing a set of key-value pairs, that are then passed into a template with {{param}} styled parameters.
Template fields within an AppSet spec are used to generate the application resource (So the application is generated by combining the Params from the generator with fields from the template)
Generators’ job is to generate these parameters and the template’s job is to consume them, and the template is then applied to a Kubernetes cluster as Argo CD Applications.

When to use an AppSet

When there's a need to

Deploy to multiple Kubernetes clusters
Deploy to different namespaces
Deploy to different namespaces on a single Kubernetes cluster
Deploy from different Git repositories or folders/branches

AppSet Example

AppSet definition with List Generator

apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
  name: guestbook
spec:
  generators:
    - list:
        elements:
          - cluster: engineering-dev
            url: 'https://kubernetes.default.svc'
          - cluster: engineering-prod
            url: 'https://kubernetes.default.svc'
  template:
    metadata:
      name: '{{cluster}}-guestbook'
    spec:
      project: default
      source:
        repoURL: 'https://github.com/argoproj/applicationset.git'
        targetRevision: HEAD
        path: 'examples/list-generator/guestbook/{{cluster}}'
      destination:
        server: '{{url}}'
        namespace: guestbook

List generator is passing the {{cluster}} and {{url}} fields into the Application template as the parameters.
These fields will be rendered as two corresponding Argo CD Applications - one for each defined cluster.

Generators

A Generator informs ApplicationSet on how to generate multiple Applications and how to deploy them.
There are currently 6 primary Generators in addition to 2 generators that can be used for combining the primary ones.

List Generator

Generates parameters based on a fixed list.
It passes key-value pairs specified within the elements section of the template.
Enables a manual approach to control the Application destination

Cluster Generator

For each registered cluster, this generator will produce params based on the list of values within the cluster secret.
These params will then be provided to the application template for each of the clusters

Git Generator

Includes 2 Sub-Types
1. Directory Generator: Generates params using directory structure of the Git repository
2. File Generator: Generates params using the content within JSON/YAML file in Git repository and reads a configuration file

Directory Generator

apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
  name: appset-demo
spec:
  generators:
    - git:
        repoURL: 'https://github.com/hseligson1/appset-demo.git'
        revision: HEAD
        directories:
          - path: examples/git-dir-generator/apps
  template:
    metadata:
      name: '{{path.basename}}'
    spec:
      project: default
      source:
        repoURL: 'https://github.com/hseligson1/appset-demo.git'
        targetRevision: HEAD
        path: '{{path}}'
      destination:
        server: 'https://kubernetes.default.svc'
        namespace: default

# Directory structure
appset-demo/examples/git-dir-generator/apps

├── app-set.yaml
├── app-1
│   ├── templates
│   ├── Chart.yaml
│   └── values.yaml
└── app-2
    ├── templates
  ├── Chart.yaml
  └── values.yaml

App name is generated based on the directory name

File Generator

apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
  name: appset-demo
spec:
  generators:
    - git:
        repoURL: 'https://github.com/hseligson1/appset-demo.git'
        revision: HEAD
        files:
          - path: examples/git-file-generator/cluster-config/**/config.json
  template:
    metadata:
      name: '{{cluster.name}}-app-1'
    spec:
      project: default
      source:
        repoURL: 'https://github.com/hseligson1/appset-demo.git'
        targetRevision: HEAD
        path: examples/git-file-generator/apps/app-1
      destination:
        server: '{{cluster.address}}'
        namespace: argocd

config.json

{
  "aws_account": "123456",
  "asset_id": "11223344",
  "cluster": {
    "owner": "cluster-admin@codefresh.com",
    "name": "environments-staging",
    "address": "https://1.2.3.4"
  }
}

Whenever a change is made to the config.json file within the cluster-config folder, it will automatically be discovered by the git-generator.

SCM Provider Generator

Used to discover and iterate over whole Git repositories (For example: whole repos of your organization)
Can be used to automate the creation of an environment/application whenever somebody creates a new repository in the organization

Pull Request Generator

Can iterate over PRs from Github and then create environments per PR (Great for creating temporary environment when a PR is created)

Cluster Decision Resource Generator

Can discover cluster based on specific K8s resources of any type that satisfy a set of criteria

Matrix/Merge Generator

Used to combine primary generators
Example: if you already have a Cluster generator for all your clusters and Git generator for all your apps, you can combine them with a matrix generator to deploy all your apps to all your clusters.

# Generates apps based on list generator
argocd app create my-application-sets \
--repo https://github.com/theJaxon/gitops-cert-level-2-examples \
--project default \
--sync-policy automatic \
--path ./application-sets/my-application-sets/ \
--dest-namespace default \
--dest-server https://kubernetes.default.svc

Git Generator example 2

apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
  name: many-apps-application-set
  namespace: argocd
spec:
  generators:
  - git:
      repoURL: https://github.com/codefresh-contrib/gitops-cert-level-2-examples.git
      revision: HEAD
      directories:
      - path: application-sets/example-apps/*
  template:      
    metadata:
      name: '{{path.basename}}'
    spec:
      # The project the application belongs to.
      project: default

      # Source of the application manifests
      source:
        repoURL: https://github.com/codefresh-contrib/gitops-cert-level-2-examples.git
        targetRevision: HEAD
        path: '{{path}}'
      
      # Destination cluster and namespace to deploy the application
      destination:
        server: https://kubernetes.default.svc
        namespace: '{{path.basename}}'

      # Sync policy
      syncPolicy:
        syncOptions:
          - CreateNamespace=true  
        automated: # automated sync by default retries failed attempts 5 times with following delays between attempts ( 5s, 10s, 20s, 40s, 80s ); retry controlled using `retry` field.
          prune: true # Specifies if resources should be pruned during auto-syncing ( false by default ).
          selfHeal: true # Specifies if partial app sync should be executed when resources are changed only in target Kubernetes cluster and no git change detected ( false by default ).

path.basename matches exactly the name of the last directory so here full path evaluates to application-sets/example-apps/prometheus in case of prometheus, basename will evaluate only to prometheus

3. Promoting Releases with GitOps

Git repository strategies

It's recommended to use environment per folder approach where all environments exist on the same branch and the filesystem has different folders that hold config files for each environment
It's NOT recommended to use ~~environment per branch~~ approach

Exercise: Environment Promotion

for namespace in {qa,staging,prod}; do
kubectl create namespace $namespace
argocd app create $namespace \
--repo https://github.com/theJaxon/gitops-cert-level-2-examples \
--project default \
--sync-policy automatic \
--path ./environment-promotion/envs/$namespace \
--dest-namespace $namespace \
--dest-server https://kubernetes.default.svc; done

Exercise: Argo Image Updater

argocd app create my-example-apps \
--repo https://github.com/theJaxon/gitops-cert-level-2-examples \
--project default \
--sync-policy automatic \
--path ./image-updater/applications \
--dest-namespace argocd \
--dest-server https://kubernetes.default.svc 

# Install argocd Image Updater
argocd app create image-updater \
--repo https://github.com/theJaxon/gitops-cert-level-2-examples \
--project default \
--sync-policy automatic \
--path ./image-updater/controller \
--dest-namespace argocd \
--dest-server https://kubernetes.default.svc

4. Common Operations

Sync Pahses/Hooks

Used when there's a specific ordering of components (Other than the default ordering provided)
Examples are:
1. Run a database migration task before the application is deployed
2. Run an email notification task after the application is deployed
3. Run a preflight check that will decide if the main sync operation will take place or not
4. Run smoke tests or some other kind of check after a deployment has taken place in order to validate it
To assign a resource to a specific phase use argocd.argoproj.io/hook annotation.
During a Sync operation, Argo CD will apply the resource during the appropriate phase of the deployment.
Hooks can be any type of Kubernetes resource kind, but tend to be Pod, Job or Argo Workflows.
Multiple hooks can be specified as a comma separated list.

Hook lifecycle and cleanup

Use argocd.argoproj.io/hook-delete-policy to decide when a hook will be deleted

Sync Waves

An alternative method for changing the order of resources
Defined using argocd.argoproj.io/sync-wave annotation
The integer value defines the ordering (Argo syncs lowest first then proceeds with higher)
Hooks and resources are assigned wave 0 by default, a wave can also be negative indicating that it should run before all other resources

Sync Ordering in Argo

The phase (First PreSync, then Sync then PostSync)
The wave they are in (lower values first)
By kind (e.g. namespaces first and then other Kubernetes resources, followed by custom resources)
By name

Exercise: Sync Hooks and Waves

k create ns example01

argocd app create example01 \
--repo https://github.com/theJaxon/gitops-cert-level-2-examples \
--project default \
--sync-policy none \
--path ./sync-hooks-waves/01-default-order \
--dest-namespace example01 \
--dest-server https://kubernetes.default.svc

k create ns example02

argocd app create example02 \
--repo https://github.com/theJaxon/gitops-cert-level-2-examples \
--project default \
--sync-policy none \
--path ./sync-hooks-waves/02-presync-job \
--dest-namespace example01 \
--dest-server https://kubernetes.default.svc 

k create ns example03

argocd app create example03 \
--repo https://github.com/theJaxon/gitops-cert-level-2-examples \
--project default \
--sync-policy none \
--path ./sync-hooks-waves/03-postsync-cleanup \
--dest-namespace example03 \
--dest-server https://kubernetes.default.svc 

k create ns example04

argocd app create example04 \
--repo https://github.com/theJaxon/gitops-cert-level-2-examples \
--project default \
--sync-policy none \
--path ./sync-hooks-waves/04-handle-sync-fail \
--dest-namespace example04 \
--dest-server https://kubernetes.default.svc 

k create ns example05

argocd app create example05 \
--repo https://github.com/theJaxon/gitops-cert-level-2-examples \
--project default \
--sync-policy none \
--path ./sync-hooks-waves/05-sync-waves \
--dest-namespace example05 \
--dest-server https://kubernetes.default.svc

k create ns example06

argocd app create example06 \
--repo https://github.com/theJaxon/gitops-cert-level-2-examples \
--project default \
--sync-policy none \
--path ./sync-hooks-waves/06-waves-and-hooks \
--dest-namespace example06 \
--dest-server https://kubernetes.default.svc

Exercies: Diff customization

argocd app create problematic-apps \
--repo https://github.com/theJaxon/gitops-cert-level-2-examples \
--project default \
--sync-policy none \
--path ./custom-diff/applications \
--dest-namespace argocd \
--dest-server https://kubernetes.default.svc

Notifications

Main components for Notification delivery

1. Triggers

Define the events and condition that will send notifications
Ex: Trigger notification when an application is synced successfully
Triggers are defined in argocd-notifications-cm

trigger.on-health-degraded: |
  - description: Application has degraded
    send:
    - app-health-degraded
    when: app.status.health.status == 'Degraded'

2. Templates

Define Notification Content
Templates are defined in Configmap called argocd-notifications-cm which is deployed in argocd namespace
argocd-notifications-secret stores tokens that will be used to access the service

email:
  subject: 'Application {{.app.metadata.name}} has been created.'
message: 'Application {{.app.metadata.name}} has been created.'
teams:
  title: 'Application {{.app.metadata.name}} has been created.'

3. Notification Services

Targets of the Notifications such as Slack, e-mail and so on

4. Subscriptions

Annotations added to ArgoCD Applications that tie everything together

# When a sync succeds send a message to slack channel1 and channel2 (Notification Services)
apiVersion: argoproj.io/v1alpha1
kind: AppProject
metadata:
  annotations:
    notifications.argoproj.io/subscribe.on-sync-succeeded.slack: my-channel1;my-channel2

theJaxon / gitops-cert-level-2-examples