Kubevirt opens a wide variety of options to bring up virtual machines on Kubernetes. This example demonstrates using KubeVirt to deploy a set of VMs across nodes. If you are on a cloud provider, you may want to run the VMs only on certain time windows to save costs. We can use Kubernetes CronJobs to schedule VMs to run during office hours and shut them down automatically after hours. For example, shut down all the VM in the dev namespace after 6 PM and bring them back up at 8 AM the next day.
We will be using KubeVirt VMPools with 3 replicas of Nginx instances. We will then define a CronJob that runs at 6 PM daily to stop the VMs and another one that runs at 8 AM to recreate it.
You should have a Kubernetes cluster with cluster-admin access, especially apiserver must have --allow-privileged=true. KubeVirt should also be installed and configured on the cluster. You may find the installation steps here.
We will define a Secret, cloud-init-with-nginx-secret.yaml to customize the VM at boot time. This config will install nginx and write a message to identify the VM. More details about cloud-init here and here
Create a namespace called dev, we will be operating from this namespace.
k create ns dev
Apply the secret:
k apply -f cloud-init-with-nginx-secret.yaml
VirtualMachinePool is a CRD from KubeVirt. Let's use this manifest vm-pool-nginx.yaml to create 3 replicas of VMs with the cloud-init config.
k apply -f vmpool-nginx.yaml
As mentioned in the VM pool manifest, it downloads the Ubuntu image from https://cloud-images.ubuntu.com/jammy/20240403/jammy-server-cloudimg-amd64.img and stores it as a data volume (a kubevirt abstraction of PVC) to attach to the VMs.
To check the status of related resources,
k get vmpool,vm,vmi,dv
NAME DESIRED CURRENT READY AGE
virtualmachinepool.pool.kubevirt.io/vmpool-nginx 3 3 3 5m
NAME AGE STATUS READY
virtualmachine.kubevirt.io/vmpool-nginx-0 5m Running True
virtualmachine.kubevirt.io/vmpool-nginx-1 5m Running True
virtualmachine.kubevirt.io/vmpool-nginx-2 5m Running True
NAME AGE PHASE IP NODENAME READY
virtualmachineinstance.kubevirt.io/vmpool-nginx-0 4m Running 10.42.2.250 kubevirt-02 True
virtualmachineinstance.kubevirt.io/vmpool-nginx-1 4m Running 10.42.0.70 kubevirt-01 True
virtualmachineinstance.kubevirt.io/vmpool-nginx-2 4m Running 10.42.1.172 kubevirt-03 True
NAME PHASE PROGRESS RESTARTS AGE
datavolume.cdi.kubevirt.io/vmpool-nginx-0 Succeeded 100.0% 1 4m
datavolume.cdi.kubevirt.io/vmpool-nginx-1 Succeeded 100.0% 1 4m
datavolume.cdi.kubevirt.io/vmpool-nginx-2 Succeeded 100.0% 1 4m
Notice that the VirtualMachineInstance (VMI)s are spread across nodes as mentioned in the VM pool manifest. Don't forget to check out the podAntiAffinity rule mentioned in the yaml.
Now, ssh to one of the vm using virtctl and check if Nginx is working as expected,
virtctl ssh ubuntu@vmpool-nginx-0
# Verify the nginx welcome page we set in the cloud-init config
ubuntu@vmpool-nginx-0:~$ curl localhost
Hello KubeVirts!!, I am being served from vmpool-nginx-0
A question would arise in mind on why the role binding is needed. Before that, we need to understand how does a pod get access to kube-api. Each pod gets a service account (usually the default service account from the namespace) token mounted which has certain RBAC permissions. Inside the pod, the token is mounted under /var/run/secrets/kubernetes.io/serviceaccount. Kubectl and virtctl cli can use this token to talk to the API server. More details regarding how to access Kubernetes API server from a Pod is explained here . So we need to ensure this service account has the required permissions to manage KubeVirt resources like VMs.
Some pointers to consider:
- A container image with kubectl and virtctl installed, using the image from here
- Service account with access to KubeVirt CRDs and rolebinding
KubeVirt already has ClusterRole called kubevirt.io:edit that allows managing VM-related objects. More details here. We can bind this role to the default service account in the current namespace:
Apply the RoleBinding for the default service account:
k apply -f kubevirt-rolebinding.yaml
If you like handling VMs from a different namespace, consider creating a ClusterRoleBinding.
So far so good. Now let's apply the CronJob stopvm-cronjob.yaml
k apply -f stopvm-cronjob.yaml
To check the newly created CronJob:
k get cj
NAME SCHEDULE SUSPEND ACTIVE LAST SCHEDULE AGE
stop-vm 0 18 * * 1-5 False 0 <none> 8s
Let's test the CronJob by manually triggering a job,
kubectl create job --from=cronjob/stop-vm stop-vm-manual
Verify the job logs and VM status,
k logs -f stop-vm-manual-k9t-dk7x5 ## use the pod name created by the job
VM vmpool-nginx-0 was scheduled to stop
VM vmpool-nginx-1 was scheduled to stop
VM vmpool-nginx-2 was scheduled to stop
# Check VM status
k get vm
NAME AGE STATUS READY
vmpool-nginx-0 1h Stopped False
vmpool-nginx-1 1h Stopped False
vmpool-nginx-2 1h Stopped False
VMs status has been changed to Stopped !!
Apply the CronJob startvm-cronjob.yaml to start the VMs at 8:00 AM every weekday:
k apply -f startvm-cronjob.yaml
To trigger the CronJob manually,
k create job --from=cronjob/start-vm start-vm-manual
# check the logs from the start-vm pod created by the job
k logs -f start-vm-manual-gcs9b
VM vmpool-nginx-0 was scheduled to start
VM vmpool-nginx-1 was scheduled to start
VM vmpool-nginx-2 was scheduled to start
# Check VM status
k get vm
NAME AGE STATUS READY
vmpool-nginx-0 1h Running True
vmpool-nginx-1 1h Running True
vmpool-nginx-2 1h Running True
If you have any questions or suggestions, please open an issue.