BigBang Environment Template + Deployer

This repo is a fork of the 'BigBang Customer Template' https://repo1.dso.mil/platform-one/big-bang/customers/template it contains the configuration and artifacts required to deploy BigBang to Kubernetes.

This repo also includes a script with a fast automated way to deploy BigBang into a cluster with a single Bash script carrying out most the repetitive tasks.

The original readme from the upstream repo is here bigbang-readme.md

Pre-reqs

Due to the way BigBang is designed and the reliance on gitops and flux there are several pre-reqs that can not be automated or scripted

You must already have a Kubernetes cluster and have kubectl configured to connect to it, otherwise you can tell the script to deploy AKS but this is optional.

1. Local Tools

Local tools & environment required are:

Bash (Linux / WSL2 / MacOS)
kubectl >= 1.21.0
gpg (sudo apt-get install -y gpg)
sops
kustomize
Azure CLI (Optional: only needed if deploying AKS)

Install scripts for all these can be obtained here https://github.com/benc-uk/tools-install

1. Accounts

Azure DevOps
Iron Bank Account

2. Set Up Git Repo

This is a set of manual pre-req steps that has to be done, and can't realistically be scripted:

Clone this repo to your machine https://azure-ecosystem.visualstudio.com/Azure%20Gov%20Engineering/_git/dsop-environment You can use your personal Azure AD account to do this. Authentication of git with Azure DevOps is outside the scope of this guide, but the git credential manager is a common approach.
Create a new branch and name it, a suggestion is to place env/ as a prefix in front of the branch name, e.g. env/dbowie , to identify each developer's own environment branch.
Push branch back to remote origin so it is tracked, e.g. git push --set-upstream origin {branch-name}
Create a set of credentials to clone the repo, these will be used by Flux, you can NOT use your Azure AD account or credentials. From the Azure DevOps page for this repo
- Click 'Clone' button again
- Click 'Generate Git Credentials' button
- Make a note of the username and password generated, they are needed for secrets.sh

3. Generate wildcard certificate for your domain

A certificate for domain in the key hostname in dev/configmap.yaml is needed for the Istio gateway.

Self signed certificate

A certificate for non-production environments can be generated by executing the following steps:

HOSTNAME=bigbang.dev
./scripts/create-root-cert.sh
./scripts/create-domain-cert.sh $HOSTNAME
ISTIO_GW_CRT=$(cat $HOSTNAME.crt | base64 -w0)
ISTIO_GW_KEY=$(cat $HOSTNAME.key | base64 -w0)

Key Vault stored certificate

If your certificate is stored already as secrets in keyvault set ISTIO_GW_CRT and ISTIO_GW_KEY to the keyvault id of those secrets in secrets.sh

export ISTIO_GW_CRT="<certificate id in keyvault>"
export ISTIO_GW_KEY="<certificate id in keyvault>"

If your certificate is stored already as secrets in keyvault set USE_KEYVAULT_CERT to true on deploy-vars.sh

export USE_KEYVAULT_CERT="true"

Changing certificate

If your certificate was changed change the value in secrets.sh and deploy-vars.sh them execute update-certs.sh

4. Configure For GitOps

Update the dev/bigbang.yaml file, and place your own branch name where it has __CHANGE_ME__

e.g.

branch: env/dbowie

Now, save and commit your change:

git add dev/bigbang.yaml
git commit -m "chore: updated git repo"
git push

This step only needs to be done once

5. Deploy

Copy secrets.sh.sample to secrets.sh and edit to with your own values and secrets as follows:

Set IRON_BANK_USER & IRON_BANK_PAT with the Username and CLI secret from your User Profile on https://registry1.dso.mil (After logging in click your username in the upper righthand corner).
Set AZDO_USER & AZDO_PASSWORD with the credentials you generated in step 2
Set ISTIO_GW_CRT & ISTIO_GW_KEY with the certificates from step 3.

Copy deploy-vars.sh.sample to deploy-vars.sh and configure as you wish

It is critical you get the values in these two files correct as they drive all the automation

If you want to deploy AKS as part of the deployment then set DEPLOY_AKS="true" by default it will not be deployed

Run the automated deployment script

cd scripts
./deploy.sh

This script will carry out the following:

One time creation of GPG keys and update to .sops.yaml if keys are found to exist, this step is skipped.
Creation/update of secrets.enc.yaml and pushed with git
OPTIONAL: Deployment of AKS cluster.
OPTIONAL: Connection to AKS cluster for kubectl etc
Creation of namespaces: bigbang and flux-system
Creation of secrets: sops-gpg, private-registry & private-git
Deployment of Flux from the main bigbang repo which will be cloned and scripts/install_flux.sh run. This can be disabled by setting DEPLOY_FLUX=false.
Removes network policies which block Flux being scraped
Deploys the dev/bigbang.yaml to the cluster
Validates the status of the deployment

Run kubectl get gitrepositories,ks,hr -A to see the status of what was just deployed.

It will take between five and ten minutes for deployment to complete and all the pods to be running and healthy.You can carry on watching the deployments with get pods -A and get helmreleases -A.

6. Configure local domain to IP address mapping

In dev, when using a domain name not recorded in a DNS server, if we want to access the virtual services created by Bigbang, we can add the IP address - domain mapping to /etc/hosts running the following commands:

# get istio gateway ip
ip=$(kubectl -n istio-system get service istio-ingressgateway -o jsonpath='{.status.loadBalancer.ingress[0].ip}')

# get domains
domains=$(kubectl --kubeconfig rke2.kubeconfig get virtualservices  -A -o jsonpath="{ .items[*].spec.hosts[*] }")

# add entry in /etc/hosts
echo "$ip $domains" | sudo tee -a /etc/hosts

Test Bigbang deployment

Tests are written in python; in order to run the follow the steps below:

Create virtual environment

python -m venv ./venv

Activate environment

source ./venv/bin/activate

Install requirements

pip install -r requirements.txt

Run tests

pytest ./tests -v

Troubleshooting

GPG Forbidden in devcontainer

Remove $HOME/.gnupg:

rm -rf $HOME/.gnupg

marlinspike / bigbang-rke2