appuv / KafkaDataGen

This repository provides Python scripts to generate simulated data and produce it into Kafka topics, facilitating testing and development of Kafka-based applications and pipelines.

#Equipment Data Generation Data Generation for the equipment stream analytics z

Prerequisite

1. Python 3.8.5
2. Docker
3. Confluent Kafka 6.2.0

Python Packages Required :

confluent_kafka,kafka,kafka-python,requests,avro

Setup

Configure the config.properties

For creating mock avro records (Equipment Mock Streams)

python equipmentProduce.py

For creating meta json records (Meta Update Streams)

python equipmentConsume.py

Schema Evolution

For schema evolution change the config.properties to match the upgraded schema

equipment-value-schema=equipment-value-upgrade.avsc
mock_data=equipment-upgrade-mock

Further Reading

Medium

Demo

Youtube