Inspired by https://github.com/kelseyhightower/kubernetes-the-hard-way
Project uses Terraform to provision infrastructure on Microsoft Azure, and then bootstraps Kubenernetes cluster from scratch by Ansible. Playbooks and roles loosely follow original project by Kelsey Hightower.
I’ve encountered some limitations based on available resources - infrastructure was provisioned on Student Subscription on Azure which is limited to some extend:
- Maximum of 6vCPU used per region.
- Maximum of 4vCPU used per Virtual Machine family.
- Very limited access to AAD (Azure Active Directory).
- Unlike GCP, there is no way of accessing Virtual Machines via ssh without public IP address without Bastion host (Recently Azure added similar option via Azure Bastion, however you still need to tunnel the traffic. For my use case, it was still easier to use plain VM as bastion host. Also cheaper.)
- Obviously, there are some differences between GCP and Azure, which might reflect on project itself.
- Single Azure VNet: 10.10.0.0/16.
- Three subnets for each compute group.
- Two resources have public IP addresses associated to them: Bastion host and Azure Load Balancer.
- Main entry point to the cluster is Standard Azure Load balancer. Currently it only accepts traffic on port 6443 (kubectl).
- Administrative access to the cluster is provided via Bastion host, which is mostly used to proxy ansible ssh connections to hosts with private addressing.
- Every subnet is secured with Network Security Group allowing only necessary traffic for given resource.
- Additionally, there is another network with range 10.20.0.0/16. This network is not allocated in Azure. Instead, this network is used by pods, and traffic to these is routed with Azure Routing Table to correct worker nodes.
- There are three groups of hosts: Bastion, Control Plane and Workers.
- Ansible group: controller
- Subnet: 10.10.10.0/24
- Pod subnet: 10.20.0.0/24
- Availability Set: kthw_controller_as
- Virtual machine type: Standard A1_v2
- Ansible group: worker
- Subnet: 10.10.20.0/24
- Pod subnet: 10.20.1.0/24
- Availability Set: kthw_worker_as
- Virtual machine type: Standard DS1_v2
- Ansible group: bastion
- Subnet: 10.10.0.0/24
- Pod subnet: 10.20.2.0/24
- Virtual machine type: Standard B1s
Project uses flat project structure, with most of the resources defined in single file.
Considering limited access to AAD on Student Subscription, project uses basic az login authentication method. Switching to other methods, like Service Principal should require no changes to the project itself.
Upon running, Terraform generates couple of files for Ansible to use, based on defined hosts:
- ansible/ssh.cfg
- ansible/inventory
- ansible/group_vars/all.yml
These files contain IP addresses which are known only after infrastructure is provisioned, so it’s logicall for Terraform to generate these files, especially since templating capabilities of Terraform are really good.
All ansible roles and playbooks follow idempotency rule.
Most of Ansible code is split into roles. They loosely follow the order in which original KTHW project follows - while they are split into respective functions, it is not as strict as usual Ansible roles should be formatted.
Playbooks are split into separate parts, which represent one area of bootstrapping the cluster, usually a single role, or similar tasks. The only one that should be used to fully bootstrap cluster is all.yml, which runs all other playbooks in correct order.