locust-todo
Simple Locust load test for my ToDo API.
This repo forms part of the work I am doing to learn Kubernetes by building applications on my Raspberry Pi Cluster.
See my blog post here: https://www.dinofizzotti.com/blog/2020-07-04-raspberry-pi-cluster-part-3-running-load-tests-with-kubernetes-and-locust/
Note: This repo is primarily for my own interests and learning. Feel free to read/copy the code, but I won't be supporting or maintaining anything here other than for my own benefit.
Locust
Locust is an open source load testing tool which allows you to write your performance tests in Python code. Test runs can be headless, or controlled via a simple web UI and it has the ability to run in a distributed mode, with multiple worker nodes controlled by a coordinator node. The core Python classes are extensible, allowing one to provide custom implementations for specific web protocols and traffic patterns.
locustfile.py
This file describes a very simple load test targetting my ToDo API, with the following steps:
- Create a ToDo item with a random description and a completion status set to “false”.
- Retrieve the ToDo item that was created in the first step.
- Update the ToDo item with a completion status set to “true.
- Delete the ToDo item.
Is this a great performance test? Not really, I don’t think this sequence of events represents meaningful user behaviour. But it should do OK for my purposes of demonstrating Locust running in my cluster - and maybe even gain some insight into the ToDo API performance.
Helm chart
To facilitate the installation of the Locust coordinator and worker nodes into my cluster I am using a Helm chart.
There does exist an existing Locust Helm chart within the official Helm repository, but it is currently referencing an older version of Locust which uses different configuration environment variables than the latest (v1.0.1) that I wish to use.
So I created my own Helm chart with the following features:
- Locust runtime parameters are specified via environment variables, configurable from the values.yml.
- Locust workers can be deployed as either a DaemonSet or as a Deployment with a configurable number of replicas.
- The former guarantees each node will run a single worker pod.
- The latter allows for the scheduler to run multiple replica worker pods per node.
- The reason for providing for both configurations is solely for my own interest in learning to work with different deployment models.
- ARM-compatible Locust image (see below).
Locust Docker build for ARM
Kubernetes sources the container images from the repository specified in the values.yaml file. Ideally I should be able to just point to the official Locust repository "locustio/locust", install my Helm chart and run my load tests. Unfortunately the Locust Docker Hub repository does not provide images built to run on the ARM architecture, and thus they will not run on my Raspberry Pi cluster. If you attempt to run an image built for x86 on an ARM device this you will most likely see an error message displaying something like exec user process caused "exec format error".
To solve this problem I cloned the Locust GitHub repository and used Docker's buildx tool to create a multi-architecture image which I then pushed to my own Docker Hub repository. No changes to the Locust code or the existing Dockerfile were required.
Running Locust in the Cluster as a DaemonSet
Node labels and nodeSelector
You will need to label your nodes accordingly. There should be at least 1x Locust coordinating node and 1x Locust worker node.
kubectl label nodes <YOUR_COORDINATING_NODE> locustRole=coordinatorThis is specified in the coordinator-deployment.yaml template:
...
nodeSelector:
locustRole: coordinator
...Likewise the remaining nodes are labelled as worker nodes:
kubectl label nodes <WORKER_NODE_1> locustRole=worker
kubectl label nodes <WORKER_NODE_2> locustRole=worker
...And the worker.yaml file specifies the following nodeSelector:
...
nodeSelector:
locustRole: worker
...Helm Chart Install
Note: I am running my applications in Kubernetes with a LoadBalancer available - you may need to make changes to the chart if you wish to use a different Service type.
To run Locust in with the DaemonSet configuration the workers.kind property in the chart values.yaml should be set to DaemonSet:
...
workers:
kind: Daemonset
...The chart can then be installed with the following command:
$ helm install --namespace locust locustchart --generate-name
NAME: locustchart-1590923674
LAST DEPLOYED: Sun May 31 12:14:34 2020
NAMESPACE: locust
STATUS: deployed
REVISION: 1
NOTES:
1. Get the application URL by running these commands:
NOTE: It may take a few minutes for the LoadBalancer IP to be available.
You can watch the status of by running 'kubectl get --namespace locust svc -w locustchart-1590923674-locust'
export SERVICE_IP=$(kubectl get svc --namespace locust locustchart-1590923674-coordinator-service-web --template "{{ range (index .status.loadBalancer.ingress 0) }}{{.}}{{ end }}")
echo http://$SERVICE_IP:80
Running Locust with worker replicas
Alternatively you may run the Locust workers in a Deployment with a number of replicas:
$ helm --namespace locust install --set workers.kind=Deployment --set workers.replicas=6 locust ./locustchart
NAME: locust
LAST DEPLOYED: Sun Jun 28 11:16:23 2020
NAMESPACE: locust
STATUS: deployed
REVISION: 1
NOTES:
1. Get the application URL by running these commands:
NOTE: It may take a few minutes for the LoadBalancer IP to be available.
You can watch the status of by running 'kubectl get --namespace locust svc -w locust-locust'
export SERVICE_IP=$(kubectl get svc --namespace locust locust-coordinator-service-web --template "{{ range (index .status.loadBalancer.ingress 0) }}{{.}}{{ end }}")
echo http://$SERVICE_IP:80