keithchambers / mantl-apps

Set of Demo Spark applications

Applications for MANTL platform

This repository contains some Spark- and Hadoop-based applications that can be run on MANTL as well as CDH platform.

Below is the full list of applications.

Benchmarking applications

Application Name	Short Description	Can be run on MANTL?	Can be run on plain CDH?
TeraSort	Spark-based TeraSort benchmark	YES, MANTL-DEMO	YES, CDH-DEMO
TestDFSIO	Spark-based TestDFSIO benchmark	YES, MANTL-DEMO	YES, CDH-DEMO

Useful applications

Application Name	Short Description	Can be run on MANTL?	Can be run on plain CDH?
Spark-based DistCP	Application to copy data from HDFS to HDFS, from Swift to HDFS and vice versa	YES, MANTL-DEMO	YES, CDH-DEMO
File Aggregator	Application to merge small files on HDFS	YES, MANTL-DEMO	YES, CDH-DEMO
Cassandra-to-HDFS	Application to download Cassandra table on HDFS	YES, MANTL-DEMO	YES, CDH-DEMO

Demo applications

Application Name	Short Description	Can be run on MANTL?	Can be run on plain CDH?
Kafka Producer	Reads directory and send messages to Kafka topic	YES, MANTL-DEMO	YES, CDH-DEMO
Kafka-to-HDFS	Reads messages from Kafka topic and writes them to HDFS	YES, MANTL-DEMO	YES, CDH-DEMO
Kafka-to-Cassandra	Reads messages from Kafka topic and writes them to Cassandra	YES	NO, requires Cassandra installation
Kafka-to-ES	Reads messages from Kafka topic and writes them to Emastic Search	YES	NO, requires ES installation
Kafka-to-Multiple	Reads messages from Kafka topic and writes them to HDFS, Cassandra and Elastic Search, runs a parallel thread that counts amount of saved records in each destination	YES, MANTL-DEMO	NO, requires Cassandra and ES installation
Kafka Monitor	Reads messages from Kafka topic on Shipped Central cluster (containing information about CPU/RAM utilization of different hosts) and writes alerts to Elastic Search if any of these metrics are beyond given thresholds	YES, MANTL-DEMO	NO, requires ES installation and is specific to ShippedCentral