tbaums / konvoy-kudo-studio

Konvoy + Kudo Studio

NB: Suggestions, constructive criticism, and (especially) PRs are most welcome!

Tested on AWS and Azure with:

• Konvoy v1.4.2
• KUDO v0.12.0
• Kafka Operator Version v1.2.0
• Zookeeper Operator Version v0.3.0

Prerequisites

1. This demo assumes you have kubectl installed and connected to a Konvoy cluster.
2. Install the KUDO CLI plugin for kubectl via: brew install kudo-cli / brew upgrade kudo-cli OR kubectl krew install kudo / kubectl krew upgrade kudo
3. Confirm you are running the latest KUDO CLI (v0.12.0) via: kubectl kudo --version
4. To execute in an airgapped environment, please follow the airgapped instructions before proceeding.

Initial setup

Import Kafka dashboard

1. Nagivate to Grafana
2. Hover over + in left-hand side nav bar
3. Select import
4. Copy and paste JSON found here
5. Click Upload
6. Select Prometheus as data source

Deploy Kudo Kafka

1. Confirm you do not have an earlier version of KUDO deployed to your cluster by running kubectl get ns kudo-system. If kubectl responds saying the namespace was not found, proceed to the next step. If you discover that KUDO is already installed in your cluster, please begin by deleting the KUDO instance on your cluster: kubectl kudo init --dry-run -o yaml | kubectl delete -f -
2. Install KUDO on your cluster: kubectl kudo init --wait
3. Next, install Zookeeper, which is a dependency for Kafka: kubectl kudo install zookeeper --wait
4. Wait for all 3 Zookeeper pods to be RUNNING and READY
5. kubectl kudo install kafka --instance=kafka -p ADD_SERVICE_MONITOR=true --wait
6. Wait for all 3 Kafka brokers to be RUNNING and READY

Deploy KUDO Cassandra

1. kubectl kudo install cassandra --instance=cassandra -p PROMETHEUS_EXPORTER_ENABLED=true --wait
2. Wait for all 3 Cassandra nodes to be RUNNING and READY

Deploy Kafka Client API, Svelte front-end, Node.js Websocket server, and Cassandra connector

1. kubectl apply -f https://raw.githubusercontent.com/tbaums/konvoy-kudo-studio/master/kafka-client-api/kafka-client-api.yaml
2. kubectl apply -f https://raw.githubusercontent.com/tbaums/konvoy-kudo-studio/master/svelte-ui/svelte-client.yaml
3. kubectl apply -f https://raw.githubusercontent.com/tbaums/konvoy-kudo-studio/master/kafka-node-js-api/kafka-node-js-api.yaml
4. kubectl apply -f https://raw.githubusercontent.com/tbaums/konvoy-kudo-studio/master/kafka-dummy-actors/kafka-dummy-actor.yaml
5. kubectl apply -f https://raw.githubusercontent.com/tbaums/konvoy-kudo-studio/master/kafka-cassandra-connector/kafka-cassandra-connector.yaml

Navigate to Svelte UI

1. Visit <your AWS elb>/svelte

Begin demonstration

Manufacturing and IoT example

1. Click 'Manufactoring & IoT' in the Nav bar
2. Explain demo architecture
3. Click '-' button to collapse architecture diagram
4. Click button 'Click me to start fetch'
5. Run kubectl scale deploy kafka-dummy-actor --replicas=1
6. Observe a single actor on the map (left) and in the actor list (on right).
7. Run kubectl scale deploy kafka-dummy-actor --replicas=7 to see the list fill in real-time and observe the actors moving around the map.

User Research example

1. Click 'User Research' in the Nav bar
2. Explain demo architecture
3. Click '-' button to collapse architecture diagram
4. Open browser 'Network' panel and reload the page. (Right click on the page, and select "Inspect Element". Then, select the 'Network' panel tab. Reload the page to start capturing network traffic.)
5. Move mouse across left-hand screenshot
6. Explain that each mouse movement captured by the browser is posted directly to the Python Kafka API server, via an endpoint exposed through Traefik
7. Observe Node.js Kafka API reading from Kafka queue and returning the mouse movements in the right-hand screenshot
8. Observe POST request duration (should be ~500ms)

Demonstrate the power of horizontal scaling

To demonstrate the power of granular microservice scaling, first we need to generate more load on the Python Kafka API. We will then observe POST request times increase. Lastly, we will scale the Python Kafka API and observe POST request times return to normal.

From User Research screen (assumes above demo steps completed):

1. kubectl scale deploy kafka-dummy-actor --replicas=70
2. Move mouse across left-hand panel
3. Observe POST request duration in browser's Network panel (should be >1000ms)
4. kubectl scale deploy kafka-client-api --replicas=5
5. Observe POST request duration (should return to ~500ms)