k8s-bench-suite

Bash scripts collection to benchmark kubernetes cluster performance

knb : Kubernetes Network Benchmark

knb is a bash script that will start a networking benchmark on a target Kubernetes cluster.

Here are some highlights:

Plain bash script with very few dependencies
Complete benchmark takes only 2 minutes
Ability to select only a subset of benchmark tests to run
Testing both TCP and UDP bandwidth
Automatic detection of CNI MTU
Includes host cpu and ram monitoring in benchmark report
Ability to create static graph images based on the result data using plotly/orca (see examples below)
No ssh access required, just an access to the target cluster through standard kubectl
No need for high privileges, the script will just launch very lightweight pods on two nodes.
Based on very lights containers images :
- infrabuilder/bench-iperf3, is used to run benchmark tests
- infrabuilder/bench-custom-monitor, is used to monitor nodes
Ability to run the whole suite in a container olegeech/k8s-bench-suite:
- Image is based on the bitnami/kubectl
- Nodes for testing can be auto-preselected

Requirements

This script needs a valid kubectl setup with an access to the target cluster.

Binaries dependencies for the host that will execute knb :

awk
grep
tail
date
kubectl
jq (for plotting)

Quickstart

Choose two nodes to act as server/client on your cluster (for example node1 and node2) . Then start the knb :

./knb --verbose --client-node node1 --server-node node2

If you omit the --verbose flag, it will also complete, but you will have no output until the end of the benchmark.

Docker quickstart

Environment variables:

NODE_AUTOSELECT Auto-selects a few nodes from cluster for running tests
MASTER_ELIGIBLE Master nodes can also be chosen

You need to mount a valid kubeconfig inside the container and provide all other required flags to knb:

docker run -e NODE_AUTOSELECT=1 -it --hostname knb --name knb --rm -v /home/user/my-graphs:/my-graphs -v /path/to/my/kubeconfig:/.kube/config olegeech/k8s-bench-suite --verbose --plot --plot-dir /my-graphs

Examples

Simple benchmark from "node1" to "node2" in verbose mode :
```
knb -v -cn node1 -sn node2
```
Benchmark from "nA" to "nB" and save data in file mybench.knbdata
```
knb -cn nA -sn nB -o data -f mybench.knbdata
```
Generate report in json from previous benchmark data file mybench.knbdata
```
knb -fd mybench.knbdata -o json
```

Plot graphs from previous benchmark data file mybench.knbdata

knb -fd mybench.knbdata --plot --plot-args '--width 900 --height 600'

To run benchmark from node A to node B, showing only result in yaml format :
```
knb -cn A -sn B -o yaml
```
To run benchmark from node Asterix to node Obelix, with the most verbose output and a result as json in a res.json file :
```
knb --debug -cn Asterix -sn Obelix -o json -f res.json
```
Running benchmark in namespace myns :
```
knb -n myns -cn node1 -sn node2
```

Run only idle and tcp benchmark :

knb -cn clientnode -sn servernode -ot idle,tcp

Usage

To display usage, use the -h flag :

aducastel@infrabuilder:~/k8s-bench-suite$ ./knb -h

knb is a network benchmark tool for Kubernetes CNI

There are two modes :
- benchmark mode : will actually run benchmark on a cluster
- from data mode : read data generated by previous benchmark with "-o data" flag

=====[ Benchmark mode ]====================================================

 Mandatory flags :

    -cn <nodename>
    --client-node <nodename>    : Define kubernetes node name that will host the client part

    -sn <nodename>
    --server-node <nodename>    : Define kubernetes node name that will host the server part

 Optionnal flags :
    -d <time-in-scd>
    --duration <time-in-scd>    : Set the benchmark duration for each test in seconds (Default 10)

    -k
    --keep                      : Keep data directory instead of cleaning it (tmp dir that contains raw benchmark data)

    -n <namespace>
    --namespace <namespace>     : Set the target kubernetes namespace

    --name <name>               : Set the name of this benchmark run

    -ot <testlist>
    --only-tests <testlist>     : Only run a subset of benchmark tests, comma separated (Ex: -ot tcp,idle)
                                  Possible values: all, tcp, udp, p2p, p2s , p2ptcp, p2pudp, p2stcp, p2sudp, idle

    -sbs <size>
    --socket-buffer-size <size> : Set the UDP socket buffer size with unit, or 'auto'. ex: '256K' (Default: auto)

    -t <time-in-scd>
    --timeout <time-in-scd>     : Set the pod ready wait timeout in seconds (Default 30)

=====[ From Data mode ]====================================================

Mandatory flags :
    -fd <path>
    --from-data <path>          : Define the path to the data to read from
                                  Data file must be rendered with '--output data'

=====[ Common optionnal flags ]============================================

    --debug                     : Set the debug level to "debug"

    -dl <level>
    --debug-level <level>       : Set the debug level
                                  Possible values: standard, warn, info, debug

    -f <filepath>
    --file <filepath>           : Set the output file

    -h
    --help                      : Display this help message

    -p
    --plot                      : Plot data using plotly/orca

    -pd
    --plot-dir                  : Directory where to save graphs
                                  Defaults to the current directory

    -pa
    --plot-args                 : Arguments to the plotly's 'orca graph' function
                                  Defaults to '--width 900 --height 500'

    -o <format>
    --output <format>           : Set the output format. Defaults to 'text'
                                  Possible values: text, yaml, json, data
    -v
    --verbose                   : Activate the verbose mode by setting debug-level to 'info'

    -V
    --version                   : Show current script version

=====[ Examples ]==========================================================

  Simple benchmark from "node1" to "node2" in verbose mode
  -------------------------------------------------------------------------
  | knb -v -cn node1 -sn node2                                            |
  -------------------------------------------------------------------------

  Benchmark from "nA" to "nB" with data saved in file "mybench.knbdata"
  -------------------------------------------------------------------------
  | knb -cn nA -sn nB -o data -f mybench.knbdata                          |
  -------------------------------------------------------------------------

  Generate report in json from previous benchmark file "mybench.knbdata"
  -------------------------------------------------------------------------
  | knb -fd mybench.knbdata -o json                                       |
  -------------------------------------------------------------------------

  Create graph images from previous benchmark file "mybench.knbdata"
  -------------------------------------------------------------------------
  | knb -fd mybench.knbdata  --plot --plot-dir ./my-graphs                |
  -------------------------------------------------------------------------

  Run only idle and tcp benchmark :
  -------------------------------------------------------------------------
  | knb -cn clientnode -sn servernode -ot idle,tcp                        |
  -------------------------------------------------------------------------

JaneLiuL / k8s-bench-suite