davidespo / kafka-load-test-env

Ephemeral Kafka Load Test Environment

Setup

Rename template.env to .env and leave it in the root directory. Add the appropriate projects, cluster names, and credentials.

AVN_TOKEN = "..."

# Primary Project, e.g. production
PROJECT_SRC = "..."

# Load Testing Project, e.g. load-test
PROJECT_DEST = "..."

# Production Cluster
KAFKA_SRC = "..."

# Load Test Cluster
KAFKA_TARGET = "..."

KAFKA_TARGET_PLAN = "business-32"
# KAFKA_TARGET_PLAN = "premium-6x-32"

KAFKA_TARGET_VERSION = "2.3"
# KAFKA_TARGET_VERSION = "2.6"

Deploy The Kafka Cluster

$ terraform init
$ terraform apply --var-file=.env .

Clone Kafka Topics

Use the provided binaries in the bin directory or run the node distribution file in dist/syncTopics.js. Note that these scripts use the .env file for api credentials.

$ bin/syncTopics-macos

Access Kafka Credentials

The following credentials are a programatic way of obtaining copies of the cluster credentials. Ensure that you use the correct service and project name for the desired cluster.

# Credential Files Only
$ avn service user-creds-download $KAFKA_SERVICE_NAME --username avnadmin --project $AVN_PROJECT
$ ls -1
ca.pem
service.cert
service.key

# Java Keystore/Truststore Utils
$ avn service user-kafka-java-creds $KAFKA_SERVICE_NAME --username avnadmin --project $AVN_PROJECT
$ ls -1
ca.pem
client.keystore.p12
client.properties
client.truststore.jks
service.cert
service.key

Shutdown

Poweroff

Cost: $0 => Powered-off services do NOT incure usage fees

Note: Preserves Credentials

This method turns off the cluster but does not remove it. That means that the credentials with remain the same between provisioning the cluster. Note that this will delete all topic meta and data. You will still need to syncTopics after restarting.

avn service update --power-off $KAFKA_SERVICE_NAME --project $AVN_PROJECT
avn service update --power-off "metrics-dashboard" --project $AVN_PROJECT
avn service update --power-off "metrics-influxdb" --project $AVN_PROJECT

# The following lines will reverse the above actions
avn service update --power-on $KAFKA_SERVICE_NAME --project $AVN_PROJECT
avn service update --power-on "metrics-dashboard" --project $AVN_PROJECT
avn service update --power-on "metrics-influxdb" --project $AVN_PROJECT

Delete

Cost: $0

This method turns completely deletes the cluster, credentials, topic meta, and all data associated with it.

$ terraform destroy --var-file=.env .
# You can reverse the following action using `terraform apply ...` to
# recreate the environment.