This repository contains a Terraform module for creating a Kubernetes cluster with Talos in the Hetzner Cloud.
- Talos is a modern OS for Kubernetes. It is designed to be secure, immutable, and minimal.
- Hetzner Cloud is a cloud hosting provider with nice terraform support and cheap prices.
Warning
It's under active development. Not all features are compatible with each other yet. Known issues are listed in the Known Issues section. If you find a bug or have a feature request, please open an issue.
Goals | Status | Description |
---|---|---|
Production ready | β | All recommendations from the Talos Production Clusters are implemented. But you need to read it carefully to understand all implications. |
Use private networks for the internal communication of the cluster | β | |
Do not expose the Kubernetes and Talos API to the public internet via Load-Balancer | β | Actually, the APIs are exposed to the public internet, but secured via the firewall_use_current_ip flag and a firewall rule that only allows traffic from one IP address. |
Possibility to change alls CIDRs of the networks | Needs to be tested. | |
Configure the Cluster as good as possible to run in the Hetzner Cloud | β | This includes manual configuration of the network devices and not via DHCP, provisioning of Floating IPs (VIP), etc. |
- A lot of information can be found directly in the descriptions of the variables.
- You can configure the module to create a cluster with 1, 3 or 5 control planes and n workers or only the control planes.
- It allows scheduling pods on the control planes if no workers are created.
- It has Multihoming configuration (etcd and kubelet listen on public and private IP).
- It uses KubePrism as cluster endpoint.
- If
cluster_api_host
is set, then you should create a corresponding DNS record pointing to either one control plane, the load balancer, floating IP, or alias IP. Ifcluster_api_host
is not set, then a record forkube.[cluster_domain]
should be created. It totally depends on your setup.
- Cilium is a modern, efficient, and secure networking and security solution for Kubernetes.
- It is used Cilium as the CNI instead of the default Flannel instead of the default Flannel.
- It provides a lot of features like Network Policies, Load Balancing, and more.
Important
The Cilium version (cilium_version
) has to be compatible with the Kubernetes (kubernetes_version
) version.
- Updates the
Node
objects with information about the server from the Cloud , like instance Type, Location, Datacenter, Server ID, IPs. - Cleans up stale
Node
objects when the server is deleted in the API. - Routes traffic to the pods through Hetzner Cloud Networks. Removes one layer of indirection.
- Watches Services with
type: LoadBalancer
and creates Hetzner Cloud Load Balancers for them, adds Kubernetes Nodes as targets for the Load Balancer.
- Applies labels to the nodes.
- Validates and approves node CSRs.
- In DaemonSet mode: CCM will use hostNetwork and current node to access kubernetes/talos API
Tip
If you don't have a Hetzner account yet, you are welcome to use this Hetzner Cloud Referral Link to claim 20β¬ credit and support this project.
- Create a new project in the Hetzner Cloud Console
- Create a new API token in the project
- You can store the token in the environment variable
HCLOUD_TOKEN
or use it in the following commands/terraform files.
Create the talos os images (ARM and x86) via packer through running the create.sh.
It is using the HCLOUD_TOKEN
environment variable to authenticate against the Hetzner Cloud API and uses the project
of the token to store the images.
The talos os version is defined in the variable talos_version
in talos-hcloud.pkr.hcl.
./_packer/create.sh
Use the module as shown in the following working minimal example:
Note
Actually, your current IP address has to have access to the nodes during the creation of the cluster.
module "terraform-hcloud-talos" {
source = "hcloud-talos/talos/hcloud"
version = "the-latest-version-of-the-module"
talos_version = "v1.7.5" # The version of talos features to use in generated machine configurations
hcloud_token = "your-hcloud-token"
# If true, the current IP address will be used as the source for the firewall rules.
# ATTENTION: to determine the current IP, a request to a public service (https://ipv4.icanhazip.com) is made.
# If false, you have to provide your public IP address (as list) in the variable `firewall_kube_api_source` and `firewall_talos_api_source`.
firewall_use_current_ip = true
cluster_name = "dummy.com"
datacenter_name = "fsn1-dc14"
control_plane_count = 1
control_plane_server_type = "cax11"
}
Or a more advanced example:
module "talos" {
source = "hcloud-talos/talos/hcloud"
version = "the-latest-version-of-the-module"
talos_version = "v1.7.5"
kubernetes_version = "1.29.7"
cilium_version = "1.15.7"
hcloud_token = "your-hcloud-token"
cluster_name = "dummy.com"
cluster_domain = "cluster.dummy.com.local"
cluster_api_host = "kube.dummy.com"
firewall_use_current_ip = false
firewall_kube_api_source = ["your-ip"]
firewall_talos_api_source = ["your-ip"]
datacenter_name = "fsn1-dc14"
control_plane_count = 3
control_plane_server_type = "cax11"
worker_count = 3
worker_server_type = "cax21"
network_ipv4_cidr = "10.0.0.0/16"
node_ipv4_cidr = "10.0.1.0/24"
pod_ipv4_cidr = "10.0.16.0/20"
service_ipv4_cidr = "10.0.8.0/21"
}
You need to pipe the outputs of the module:
output "talosconfig" {
value = module.talos.talosconfig
sensitive = true
}
output "kubeconfig" {
value = module.talos.kubeconfig
sensitive = true
}
Then you can then run the following commands to export the kubeconfig and talosconfig:
terraform output --raw kubeconfig > ./kubeconfig
terraform output --raw talosconfig > ./talosconfig
Move these files to the correct location and use them with kubectl
and talosctl
.
kubelet_extra_args = {
system-reserved = "cpu=100m,memory=250Mi,ephemeral-storage=1Gi"
kube-reserved = "cpu=100m,memory=200Mi,ephemeral-storage=1Gi"
eviction-hard = "memory.available<100Mi,nodefs.available<10%"
eviction-soft = "memory.available<200Mi,nodefs.available<15%"
eviction-soft-grace-period = "memory.available=2m30s,nodefs.available=4m"
}
sysctls_extra_args = {
# Fix for https://github.com/cloudflare/cloudflared/issues/1176
"net.core.rmem_default" = "26214400"
"net.core.wmem_default" = "26214400"
"net.core.rmem_max" = "26214400"
"net.core.wmem_max" = "26214400"
}
kernel_modules_to_load = [
{
name = "binfmt_misc" # Required for QEMU
}
]
- Changes in the
user_data
(e.g.talos_machine_configuration
) andimage
(e.g. version upgrades withpacker
) will not be applied to existing nodes, because it would force a recreation of the nodes.
enable_alias_ip
can lead to error messages occurring during the first bootstrap. More about this here: siderolabs/talos#8493 If these error messages occur, one control plane must be restarted after complete initialisation once. This should resolve the error.- IPv6 dual stack is not supported by Talos yet. You can activate IPv6 with
enable_ipv6
, but it should not have any effect. enable_kube_span
let's the cluster not get in ready state. It is not clear why yet. I have to investigate it.
- kube-hetzner For the inspiration and the great terraform module. This module is based on many ideas and code snippets from kube-hetzner.
- Talos For the incredible OS.
- Hetzner Cloud For the great cloud hosting.