mateuszdyminski / am-pipeline

Simple data pipeline showing how to create full text search over some dataset

AM pipeline

Simple data pipeline showing how to create full text search over some dataset.

Description

We are using Kafka and Elasticsearch.

There are 4 components of AM pipeline:

• Feeder - reads CSV file with users as input and puts them into Kafka
• Indexer - reads users from Kafka and put them in Elasticsearch
• Kafka-Streams - reads users from Kafka and calculates statistics on top of them
• Analyzer - UI which provides users visualization
  • locations of users on the map
  • search for any particular field or search over all fields
  • pagination of results

Kafka Presentation

Please install [[https://godoc.org/golang.org/x/tools/present][https://godoc.org/golang.org/x/tools/present]]

Then run:

cd presentation && present

Requirements

• minikube
• installed kubectl
• go >= 1.8
• yarn - to install JS requirements
• maven (for kafka streams)
• java jdk >= 8

Infrastructure

To start all services

Read README.md file in infra/kube directory