Stock Data Processing with Spring Cloud Stream and the Confluent Platform
- Check the docker bridge IP (
docker network inspect bridge
) and change the URLs in theapplication.yml
config file and in theKAFKA_ADVERTISED_LISTENERS
url in thedocker-compose.yml
files if needed - Download the Stock Data CSVs
cd cp-docker ./download-data.sh
- Start the docker containers with Kafka, Zookeeper, Kafka Connect and the Schema Registry.
docker-compose up -d docker-compose ps
- When all containers are up, create a new connect task for the CSV import to a Kafka Topic.
The connect plugin will register the schema of the imported data in the Schema registry../start-connector.sh curl localhost:8083/connector-plugins | jq
- The connector will send messages to Kafka when CSVs are placed in
cp-docker/connect/spool-input
head -n 10 us-etfs.csv > connect/spool-input/etfs.csv
- When the application is started, it should receive the messages of the first few rows. The Messages are sent in Avro format and are deserialized to pojos, which were created with the Avro maven plugin
- Confluent Schema Registry: https://docs.confluent.io/current/schema-registry/docs/intro.html
- Kafka Connect Plugin for CSV import: https://github.com/jcustenborder/kafka-connect-spooldir
- Spring Cloud Stream Reference: https://docs.spring.io/spring-cloud-stream/docs/Elmhurst.M3/reference/htmlsingle/
- Spring Cloud Stream Kafka Binder: https://docs.spring.io/autorepo/docs/spring-cloud-stream-binder-kafka-docs/1.1.0.M1/reference/htmlsingle/
- Kafka Streams: https://kafka.apache.org/documentation/streams/