Synthetic avro message generator
- Creates new Kafka topics with particular number of
partitionsandreplication - Registers Avro schemas into the schema-registry
- Generates sets of i.e. 1 Million Avro messages of particular specifications
The following topics are populated
| Kafka Topic | Avro messages | Versions |
|---|---|---|
| generator-text | Single field text with 50 to 100 chars payload |
1 |
| generator-types | Basic Avro types string, boolean, int, long, float, double |
1 |
| generator-types-upsert | As above, but 1% of the messages contain the same value in field text |
1 |
| generator-sql | Contains reserved SQL words as fields and also in contents |
1 |
| generator-shipments | Item shipments to stores for e-commerce use-case | 1 |
| generator-sales | Item sales for e-commerce use-case | 1 |
| generator-evolution-widen | Testing Avro Evolution type widenning ¹ |
2 |
| generator-evolution-add | Testing Avro Evolution adding new fields with default value ² |
4 |
¹ float is widenned to double both old and new records are generated
² Keeps adding new fields
mvn clean install
Export 3 system variables
export BROKERS="broker-hostname:9092"
export ZK="zookeeper-hostname:2181"
export SCHEMA_REGISTRY="http://schema-registry-hostname:8081"
And then execute - by passing in number of messages per set and partitions
cd target
./landoop-avro-generator 1000000 10
kafka-topics --delete --zookeeper $ZK --topic generator-text
kafka-topics --delete --zookeeper $ZK --topic generator-types
kafka-topics --delete --zookeeper $ZK --topic generator-types-upsert
kafka-topics --delete --zookeeper $ZK --topic generator-sql
kafka-topics --delete --zookeeper $ZK --topic generator-shipments
kafka-topics --delete --zookeeper $ZK --topic generator-sales
kafka-topics --delete --zookeeper $ZK --topic generator-evolution-widen
kafka-topics --delete --zookeeper $ZK --topic generator-evolution-add
sleep 2
kafka-topics --zookeeper $ZK --list | grep "generator-"
Delete the _schemas topic
kafka-topics --delete --zookeeper $ZK --topic _schemas
kafka-run-class kafka.tools.ZooKeeperMainWrapper -server $ZK delete /schema_registry/schema_id_counter
kafka-avro-console-consumer --zookeeper $ZK --topic _schemas --from-beginning