xunleii / terraform-module-k3s

Terraform module to manage a k3s cluster on given machines

terraform-module-k3s

Terraform module to create a k3s cluster with multi-server and annotations/labels/taints management features.

⚠️ Security disclosure

Because the use of external references on the destroy provisioner is deprecated by Terraform, storing information inside each resource is mandatory in order to manage several functionalities such as automatic node draining and field management. As a result, several fields such as the connection block will be available in your TF state. This means that the password or private key used will be clearly readable in this TF state.
Please be very careful to store your TF state securely if you use a private key or password in the connection block.

Example (based on Hetzner Cloud example)

module "k3s" {
  source = "xunleii/k3s/module"

  depends_on_    = hcloud_server.agents
  k3s_version    = "latest"
  cluster_domain = "cluster.local"
  cidr = {
    pods     = "10.42.0.0/16"
    services = "10.43.0.0/16"
  }
  drain_timeout  = "30s"
  managed_fields = ["label", "taint"] // ignore annotations

  global_flags = [
    "--flannel-iface ens10",
    "--kubelet-arg cloud-provider=external" // required to use https://github.com/hetznercloud/hcloud-cloud-controller-manager
  ]

  servers = {
    for i in range(length(hcloud_server.control_planes)) :
    hcloud_server.control_planes[i].name => {
      ip = hcloud_server_network.control_planes[i].ip
      connection = {
        host        = hcloud_server.control_planes[i].ipv4_address
        private_key = trimspace(tls_private_key.ed25519_provisioning.private_key_pem)
      }
      flags = [
        "--disable-cloud-controller",
        "--tls-san ${hcloud_server.control_planes[0].ipv4_address}",
      ]
      annotations = { "server_id" : i } // theses annotations will not be managed by this module
    }
  }

  agents = {
    for i in range(length(hcloud_server.agents)) :
    "${hcloud_server.agents[i].name}_node" => {
      name = hcloud_server.agents[i].name
      ip   = hcloud_server_network.agents_network[i].ip
      connection = {
        host        = hcloud_server.agents[i].ipv4_address
        private_key = trimspace(tls_private_key.ed25519_provisioning.private_key_pem)
      }

      labels = { "node.kubernetes.io/pool" = hcloud_server.agents[i].labels.nodepool }
      taints = { "dedicated" : hcloud_server.agents[i].labels.nodepool == "gpu" ? "gpu:NoSchedule" : null }
    }
  }
}

Inputs

Name	Description	Type	Default	Required
servers	K3s server nodes definition. The key is used as node name if no name is provided.	map(any)	n/a	yes
agents	K3s agent nodes definitions. The key is used as node name if no name is provided.	map(any)	{}	no
cidr	K3s network CIDRs (see https://rancher.com/docs/k3s/latest/en/installation/install-options/).	object({ pods = string services = string })	{ "pods": "10.42.0.0/16", "services": "10.43.0.0/16" }	no
cluster_domain	K3s cluster domain name (see https://rancher.com/docs/k3s/latest/en/installation/install-options/).	string	"cluster.local"	no
depends_on_	Resource dependency of this module.	any	null	no
drain_timeout	The length of time to wait before giving up the node draining. Infinite by default.	string	"0s"	no
generate_ca_certificates	If true, this module will generate the CA certificates (see k3s-io/k3s#1868 (comment)). Otherwise rancher will generate it. This is required to generate kubeconfig	bool	true	no
global_flags	Add additional installation flags, used by all nodes (see https://rancher.com/docs/k3s/latest/en/installation/install-options/).	list(string)	[]	no
k3s_install_env_vars	map of enviroment variables that are passed to the k3s installation script (see https://docs.k3s.io/reference/env-variables)	map(string)	{}	no
k3s_version	Specify the k3s version. You can choose from the following release channels or pin the version directly	string	"latest"	no
kubernetes_certificates	A list of maps of cerificate-name.[crt/key] : cerficate-value to copied to /var/lib/rancher/k3s/server/tls, if this option is used generate_ca_certificates will be treat as false	list( object({ file_name = string, file_content = string }) )	[]	no
managed_fields	List of fields which must be managed by this module (can be annotation, label and/or taint).	list(string)	[ "annotation", "label", "taint" ]	no
name	K3s cluster domain name (see https://rancher.com/docs/k3s/latest/en/installation/install-options/). This input is deprecated and will be remove in the next major release. Use cluster_domain instead.	string	"!!!DEPRECATED!!!"	no
separator	Separator used to separates node name and field name (used to manage annotations, labels and taints).	string	"|"	no
use_sudo	Whether or not to use kubectl with sudo during cluster setup.	bool	false	no

Outputs

Name	Description
kube_config	Genereated kubeconfig.
kubernetes	Authentication credentials of Kubernetes (full administrator).
kubernetes_cluster_secret	Secret token used to join nodes to the cluster
kubernetes_ready	Dependency endpoint to synchronize k3s installation and provisioning.
summary	Current state of k3s (version & nodes).

Providers

Name	Version
http	~> 3.0
null	~> 3.0
random	~> 3.0
tls	~> 4.0

Frequently Asked Questions

How to customise the generated kubeconfig

It is sometimes necessary to modify the context or the cluster name to adapt kubeconfig to a third-party tool or to avoid conflicts with existing tools. Although this is not the role of this module, it can easily be done with its outputs :

module "k3s" {
  ...
}

local {
  kubeconfig = yamlencode({
    apiVersion      = "v1"
    kind            = "Config"
    current-context = "my-context-name"
    contexts = [{
      context = {
        cluster = "my-cluster-name"
        user : "my-user-name"
      }
      name = "my-context-name"
    }]
    clusters = [{
      cluster = {
        certificate-authority-data = base64encode(module.k3s.kubernetes.cluster_ca_certificate)
        server                     = module.k3s.kubernetes.api_endpoint
      }
      name = "my-cluster-name"
    }]
    users = [{
      user = {
        client-certificate-data : base64encode(module.k3s.kubernetes.client_certificate)
        client-key-data : base64encode(module.k3s.kubernetes.client_key)
      }
      name : "my-user-name"
    }]
  })
}

License

terraform-module-k3s is released under the MIT License. See the bundled LICENSE file for details.

Generated with ❤️ by terraform-docs