This repository contains scripts, manifests and IaC to create standalone k8s clusters on AWS and GCP. For Cluster Mesh and Cluster API clusters check out olga-mir/k8s-multi-cluster
The clusters are intended for learning and exploration.
These clusters are not hardened for security, primarily for lack of time and the fact that they are short-lived, bootstrapped in learning projects with hard limits. Nodes and API server are exposed with public IP. But SSH and API server access is allowed only from the same IP address that it was created from
- AWS (kOps): k8s - 1.28.3; kOps - 1.28.0
- AWS (EKS): k8s - 1.27, eksctl - 0.164.0
- GCP (GKE): k8s - 1.23.13
.
├── LICENSE
├── README.md
├── aws
│ ├── Makefile
│ ├── Makefile.preflight
│ ├── README.md
│ ├── eksctl // EKS cluster with `eksctl`
│ ├── foundation // network and IAM
│ └── kops // AWS non-EKS cluster
├── docs
│ └── setup.md // Tools setup and general info
├── gcp
│ ├── README.md
│ ├── crossplane // TODO
│ ├── gcloud // Create GKE cluster with `gcloud`
│ ├── multi-network-pods // https://cloud.google.com/kubernetes-engine/docs/how-to/setup-multinetwork-support-for-pods
│ └── terraform // Create GKE cluster with terraform
└── scripts // Mostly scripts to install tools on nodes for debug, low level Linux digging, performance engineering and eBPF
Both GCP and AWS folders contain static resources (VPC, IAM, Buckets) that usually don't require cleanup because they don't incur any cost. Beware that VPC stacks may include NATs or other resources that do incur cost and therefore have to be cleaned after each session
Detailed instructions provided in ./aws/README.md
Tech: kOps, jsonnet, Makefile, aws cli
make kops-create-cluster
Cleanup:
make kops-delete-cluster-yes
tech: eksctl, aws cli, cloudformation
Deploy: ./aws/eksctl/create.sh
Cleanup: ./aws/eksctl/cleanup.sh
Tools: gcloud, terraform, Makefile
Currently not all variables have been abstracted away and the tf code has hardcoded values for my VPC, but it does create a valid GKE cluster when these values are set correctly.
cd gcp/terraform
envsubst < foundation/template-tfvars > foundation/dev.tfvars
envsubst < template-tfvars > dev.tfvars
To create a VPC (only needed once):
make create-vpc
To create a cluster:
make create-gke
This will create tf plan if it doesn't exist or if dependencies changed
To delete the cluster:
make cleanup
To delete the cluster and supporting resources:
make full-cleanup
Demo provisioning management cluster (in bootstrap and pivot pattern) with workload clusters that are synced by Flux is available in my other repo: https://github.com/olga-mir/k8s-multi-cluster These are AWS clusters (not EKS) running Cilium in kube-proxy-free mode, with cluster-mesh coming soon.
% k get cluster -A
NAMESPACE NAME PHASE AGE VERSION
cluster-01 cluster-01 Provisioned 12m
cluster-02 cluster-02 Provisioning 60s
cluster-mgmt cluster-mgmt Provisioned 13m