TheKeyholdingCompany / message-producer

Kafka Broadcaster

This is a simple Kafka broadcaster that reads from a CSV of events and a json template and sends the content to a Kafka topic. It can also print out events sent to a topic when in listening mode.

This was originally developed by a group of 5 students doing a placement at The Keyholding Company through Next Tech Girls:

• Antonina Basaraba
• Dila Sisek
• Lasya Sura
• Nicola Goh
• Millie Neale

Usage

Populate events.csv with the information you want to use and template.json with how you want the events to look like when sent to Kafka.

Run this project using python:

cd ./app
python3 main.py send

See python3 main.py --help for more information.

Example

Take this events.csv:

name,age,occupation,city
nicola,17,student,London
millie,15,student,London
antonina,15,student,London
dila,15,student,London
lasya,15,student,London

The header of the CSV will determine which placeholders are available to be replaced in template.json. In this case, the placeholders are name, age, occupation and city:

{
    "name": "{name}",
    "age": "{age}",
    "occupation": "{occupation}",
    "city": "{city}",
    "department": "Technology",
    "type": "NewPerson"
}

Kafka Broadcaster will send 5 events to the topic:

{ "name": "nicola", "age": "17", "occupation": "student", "city": "London", "department": "Technology", "type": "NewPerson" }

{ "name": "millie", "age": "15", "occupation": "student", "city": "London", "department": "Technology", "type": "NewPerson" }

{ "name": "antonina", "age": "15", "occupation": "student", "city": "London", "department": "Technology", "type": "NewPerson" }

{ "name": "dila", "age": "15", "occupation": "student", "city": "London", "department": "Technology", "type": "NewPerson" }

{ "name": "lasya", "age": "15", "occupation": "student", "city": "London", "department": "Technology", "type": "NewPerson" }