tigstep's repositories
kafka_storm_pipeline
A real-time, dockerized Kafka event processor pipeline (utilizing an Apache Storm topology). The project is running in AWS with automated infrastructure deployment and execution, using Ansible.
ecs_airflow
A project that sets up an AWS ECS infrastructure and hosts an Apachi Airflow running inside a docker container.
avro_sample_serde
A sample project that AVRO serializes and deserializes a record. Some logic from this will later be used in tigstep/KafkaConnectSpooldir_Avro_Kafka.
docker_kafka_cluster
This is the upgraded version of gten/docker-kafka-cluster (kafka version is changed from 0.9 to 2.1.0 and the start-kafka.sh is changed to work with kafka2.1.0 specific server.properties)
file_obfuscator_importer_pipeline
A simple file obfuscation/ingestion pipeline that runs on AWS with automated infrastructure deployment and execution.
kafka_connect_spooldir_pipeline
A dockerized Confluent Kafka Cluster running on AWS EC2 instances that uses the Spooldir Connector(https://github.com/jcustenborder/kafka-connect-spooldir) to spool a directory, Avro serialize the data using the Schema Registry and publish to Kafka Brokers. This project also makes the Confluent Control Center available for visualization.
kafka_misc
This repo containes various modules/classes related to Apache Kafka development efforts
visitor_count
A sample Spark exercise to count the visitors based on given conditions.