This tutorial is a simple step by step instruction to reach the following goal:
- Install a on-prem Apache Kafka
- Configure the Apache Kafka tool mirrormaker
- Send message produced on-prem to Microsoft Azure Event Hub
Please note: This tutorial is only for dev / poc (proof of concept) environment and not for production environment.
Prerequisite:
- Ubuntu 20.04 (on Windows 10 with WSL 2)
- MS Visual Studio Code
- Microsoft Azure subscription
Install java on Ubuntu 20.04:
$ sudo apt update
$ sudo apt install openjdk-8-jdk -y
$ java -version
$ export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
$ export PATH=$PATH:$JAVA_HOME/bin
Install Apache Kafka on Ubuntu 20.04:
$ mkdir kafka
$ cd kafka
$ wget https://downloads.apache.org/kafka/2.7.0/kafka_2.13-2.7.0.tgz
$ tar -zxf kafka_2.13-2.7.0.tgz
$ sudo mv kafka_2.13-2.7.0 /usr/local/kafka
$ sudo mkdir /tmp/kafka-logs
$ sudo mkdir /tmp/zookeeper
Start Apache Zookeeper on Ubuntu 20.04:
$ sudo /usr/local/kafka/bin/zookeeper-server-start.sh /usr/local/kafka/config/zookeeper.properties
Start Apache Kafka Server on Ubuntu 20.04:
$ sudo /usr/local/kafka/bin/kafka-server-start.sh /usr/local/kafka/config/server.properties
Create Kafka topic on Ubuntu 20.04:
$ /usr/local/kafka/bin/kafka-topics.sh --create --topic quickstart-events --bootstrap-server localhost:9092
$ /usr/local/kafka/bin/kafka-topics.sh --create --topic mymachine --bootstrap-server localhost:9092
$ /usr/local/kafka/bin/kafka-topics.sh --list --bootstrap-server localhost:9092
Microsoft Azure Event-Hub:
Go to your Microsoft Azure subscription.
Create a resource
Create
You have to create a "Standard" tier Event Hubs namespace: In this case the Kafka endpoint for the namespace is automatically enabled.
Next: Features > Next: Tags > Next: Review + create > Create
Go to resource
Click on
and copy the "Connection string–primary key"
e.g. Endpoint=sb://kafkaazure.servicebus.windows.net/;SharedAccessKeyName=RootManageSharedAccessKey;SharedAccessKey=[your key]
for later use.
Configure MirrorMaker on Ubuntu 20.04:
$ cd /usr/local/kafka/config
$ code .
Visual Studio Code opens. Create File / New File and enter the following configuration:
bootstrap.servers=localhost:9092
exclude.internal.topics=true
client.id=mirror_maker_consumer
group.id=mirror_maker_consumer
File / Save as sourceCluster1Consumer.config
Create File / New File and enter the following configuration:
bootstrap.servers=kafkaazure.servicebus.windows.net:9093
security.protocol=SASL_SSL
sasl.mechanism=PLAIN
sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required username="$ConnectionString" password="Endpoint=sb://kafkaazure.servicebus.windows.net/;SharedAccessKeyName=RootManageSharedAccessKey;SharedAccessKey=[your key];acks=1
batch.size=50
client.id=mirror_maker_test_producer
File / Save as azureClusterProducer.config
Start MirrorMaker on Ubuntu 20.04:
$ sudo /usr/local/kafka/bin/kafka-mirror-maker.sh --consumer.config /usr/local/kafka/config/sourceCluster1Consumer.config --num.streams 1 --producer.config /usr/local/kafka/config/azureClusterProducer.config --whitelist=".*"
Start Kafka producer on Ubuntu 20.04:
$ sudo /usr/local/kafka/bin/kafka-console-producer.sh --topic mymachine --bootstrap-server localhost:9092
Enter the following value:
{"sensor_id": 1,"ltime": 1613204245,"temp": 5.5,"status": 1}
Start Kafka consumer on Ubuntu 20.04:
$ sudo /usr/local/kafka/bin/kafka-console-consumer.sh --topic mymachine --from-beginning --bootstrap-server localhost:9092
Microsoft Azure Event-Hub:
In the Event Hubs Namespace go to Event Hubs:
An Event Hub with the name mymachine ist generated automatically. Click on the mymachine Event Hub:
Go to mymachine: Your first on-prem produced Kafka message has been transmitted to Microsoft Azure as you can see for example in "Messages".
Please note that it can take a short instance for the messages to appear in the dashboard below.
