sammyas / charmander

Charmander Scheduler Lab - Mesos, Docker, cAdvisor, InfluxDB, Spark

Charmander Scheduler Lab

Charmander is a lab environment for measuring and analyzing resource-scheduling algorithms.

The project got started in Summer 2014 by Theodora Chu as an internship project. It was motivated by a paper from Stanford University: "Quasar: Resource-Efficient and QoS-Aware Cluster Management".

Charmander at its core provides an easy to use environment to a) schedule and deploy compute-tasks in a multi-node setup and b) measure the corresponding cpu, memory, and network-loads.

The collected measurements can afterwards be analyzed using the included Spark analytics workbench and subsequently those results can be fed back in to the scheduler.

Obviously this lab-setup can be used for other use-cases like testing and analyzing machine-learning based anomaly-detection, noisy-neighbor detection, or profiling algorithms, or simply serving as the load-pattern verification authority in a continuous integration environment.

Setup and Run Charmander

All that is required to run a simple lab setup and an experiment is Vagrant, VirtualBox, curl, a fast internet connection, and a bit of spare time.

All the steps are automated and are part of simple scripts that come with the Charmander project. All the additional software needed during the setup gets installed and built inside the VMs, nothing additionally gets installed on the host itself.

1. Setup Nodes

   • Configure and build the different nodes with Vagrant and VirtualBox
   • Reload and reset environment

2. Build and deploy Scheduler

   • Clone the Charmander Scheduler projects
   • Compile it inside the master node
   • Deploy and run it

3. Build and deploy Load-Simulators

   • Build the Docker images for the different load-simulators on all the lab nodes

4. Build and deploy Analytics Stack

   • Build all the Docker images for the full analytics stack on the analytics node

5. Build and run a simple Experiment, maxusage

   • Build and run a simple experiment that changes the resource allocation based on insights from previous runs

6. Spark Analytics - Build your own Experiment

   • What are the extension point to build your own Experiment
   • How to run and use Spark with Charmander
   • How to build and use iPython and Spark-Kernel, experimental!

7. Scheduler API

   • List of all the different Scheduler REST APIs

8. Script Reference

   • List of all the include scripts/tools

9. Tips and Tricks

   • Some tips and tricks for common issues

The different Charmander projects on GitHub

1. github.com/att-innovate/charmander

   The main project, this project, it contains all the scripts to set up the lab.

2. github.com/att-innovate/charmander-scheduler

   Charmander Scheduler is our Framework for Mesos.

3. github.com/att-innovate/charmander-heapster

   A slimmed-down version of Google's Heapster project.

4. github.com/att-innovate/charmander-spark

   This project contains Charmander-specific helper functions for Spark.

Additional thanks goes to

some additional open source projects and a blog post that have inspired us:

• VoltFramework
• Mesos-Go
• A Docker Dev Environment in 24 Hours!

All product and company names are trademarks™ or registered® trademarks of their respective holders. Use of them does not imply any affiliation with or endorsement by them.

Pokémon character names are trademarks of Nintendo. Other trademarks are the property of their respective owners.