RKE on OpenStack
This repo scratches my own itch when it comes to deploying Kubernetes using RKE on OpenStack.
You'll need a fairly standard OpenStack deployment with LBaaS via Octavia. The code in this repo assumes that it's a "public" OpenStack cloud with an EC2-esque network configuration in which instances are provisioned on a private network and are allocated floating IP addresses where necessary.
Block storage via Cinder isn't currently supported.
Pre-requisites
Edit terraform.tfvars
and specify the name of your SSH keypair and your OpenStack API password. You can also increase the number of controller nodes and worker nodes if necessary. Note that controller nodes have to be an odd number, as these run etcd and an odd number is required in order to establish cluster quorum.
Install the required Terraform providers as dependencies:
$ terraform init
You'll need to manually (for now) install the RKE provider by following the instructions here: https://github.com/rancher/terraform-provider-rke
Deploying
$ terraform plan -out plan.out
[..]
Plan: 26 to add, 0 to change, 0 to destroy.
------------------------------------------------------------------------
This plan was saved to: plan.out
To perform exactly these actions, run the following command to apply:
terraform apply "plan.out"
$ terraform apply plan.out
module.rancher-network.openstack_networking_network_v2.rancher_network: Creating...
module.rke-masters.openstack_networking_floatingip_v2.instance_fip[0]: Creating...
module.rancher-network.openstack_networking_router_v2.rancher_router: Creating...
module.rke-workers.openstack_networking_floatingip_v2.instance_fip[0]: Creating...
[..]
Apply complete! Resources: 26 added, 0 changed, 0 destroyed.
Deploying a cluster should take somewhere between 5-10 minutes. Once your cluster has successfully deployed, you can source the generated kubeconfig file by doing:
$ export KUBECONFIG=$(pwd)/kube_config_cluster.yml
And verify that Kubernetes is OK:
$ kubectl get nodes
NAME STATUS ROLES AGE VERSION
sausages-controlplane-etcd-instance1 Ready controlplane,etcd 2m49s v1.17.4
sausages-worker-instance1 Ready worker 2m41s v1.17.4
Now wait for the various system-related pods and jobs to run to completion and become ready:
$ kubectl get pods -A
NAMESPACE NAME READY STATUS RESTARTS AGE
ingress-nginx default-http-backend-67cf578fc4-9h7hv 1/1 Running 0 2m27s
ingress-nginx nginx-ingress-controller-dsvxt 1/1 Running 0 2m27s
kube-system canal-9wdrq 2/2 Running 0 3m14s
kube-system canal-qj8pm 2/2 Running 0 3m14s
kube-system coredns-7c5566588d-klpmr 1/1 Running 0 3m2s
kube-system coredns-autoscaler-65bfc8d47d-ffvqh 1/1 Running 0 3m1s
kube-system metrics-server-6b55c64f86-9pckk 1/1 Running 0 2m54s
kube-system rke-coredns-addon-deploy-job-9d2hg 0/1 Completed 0 3m8s
kube-system rke-ingress-controller-deploy-job-nwtcw 0/1 Completed 0 2m46s
kube-system rke-metrics-addon-deploy-job-zl4dn 0/1 Completed 0 2m58s
kube-system rke-network-plugin-deploy-job-24qv9 0/1 Completed 0 3m19s
At this point your cluster is ready to use.
Post-deployment
The deployment code in this example configures the in-tree OpenStack cloud controller. This means that any services you create of type LoadBalancer to expose your application will automatically create an OpenStack Octavia loadbalancer and allocate a public IP address. You can test this functionality by doing the following:
$ kubectl run echoserver --image=gcr.io/google-containers/echoserver:1.10 --port=8080
pod/echoserver created
$ cat <<EOF | kubectl apply -f -
---
kind: Service
apiVersion: v1
metadata:
name: loadbalanced-service
spec:
selector:
run: echoserver
type: LoadBalancer
ports:
- port: 80
targetPort: 8080
protocol: TCP
EOF
service/loadbalanced-service created
Now verify that the service we've created has an External IP allocated. This might stay in <pending>
for a brief while until the Octavia loadbalancer is available:
$ kubectl get service loadbalanced-service
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
loadbalanced-service LoadBalancer 10.43.154.100 193.16.42.81 80:32023/TCP 2m7s
And to test:
$ curl 193.16.42.81
Hostname: echoserver
[..]
Stateful workloads
If you want to deploy stateful workloads with a volume resource definition (for example when defining a StatefulSet) then you need to add the local-path-provisioner:
$ kubectl apply -f https://raw.githubusercontent.com/rancher/local-path-provisioner/master/deploy/local-path-storage.yaml
namespace/local-path-storage created
serviceaccount/local-path-provisioner-service-account created
clusterrole.rbac.authorization.k8s.io/local-path-provisioner-role created
clusterrolebinding.rbac.authorization.k8s.io/local-path-provisioner-bind created
deployment.apps/local-path-provisioner created
storageclass.storage.k8s.io/local-path created
configmap/local-path-config created
$ kubectl annotate storageclass --overwrite local-path storageclass.kubernetes.io/is-default-class=true
storageclass.storage.k8s.io/local-path annotated
Troubleshooting
If your Pods sit in pending, it could be because the cloud controller manager hasn't removed the NoSchedule taint on your worker nodes. If it's created the loadbalancer OK but no pods are being scheduled, go ahead and remove the taint:
$ kubectl taint node --selector='!node-role.kubernetes.io/master' node.cloudprovider.kubernetes.io/uninitialized=true:NoSchedule-
node/sausages-controlplane-etcd-instance1 untainted
node/sausages-worker-instance1 untainted
Your Pods should move to ContainerCreating
.