jamster / emr_scripts

Some useful scripts for Amazon's Elastic Map Reduce

EMR Scripts

Useful scripts to help you get all set up with EMR, such as bootstrap actions

• Installing Ganglia <ganglia/install_ganglia.rb>
• Installing Screen and some other utilities <config/install_screen_tree_and_config_s3.sh>

Install Ganglia

Place the script somewhere in you S3 buckets... I keep a bucket for bootstrap-actions.

I use the run-if script written by the EMR team to determine whether or not i am installing the master or slave version of Ganglia.

Then when launching the cluster with the EMR client, add the following bootstrap actions:

--bootstrap-action s3://elasticmapreduce/bootstrap-actions/run-if --args "instance.isMaster=true,s3://<your-emr-bucket>/bootstraps/install_ganglia.rb,master" \
--bootstrap-action s3://elasticmapreduce/bootstrap-actions/run-if --args "instance.isMaster!=true,s3://<your-emr-bucket>/bootstraps/install_ganglia.rb,slave" \

Modify as needed

Install Screen, Tree, and configure S3

This can obviously be broken out if needed, but was simple enough to just do in one file

Use as:

--bootstrap-action s3://<your-emr-bucket>/bootstraps/install_screen_tree_and_config_s3.sh

Simple as that.

Just don't forget to replace your ACCESS_KEY AND SECRET_KEY AND PASSWORD

TODO

Abstract this out for other cloud services and test on EC2 without EMR

Authors

Jason Amster [e-mail]

Acknowledgments

install_ganglia.rb

This code was brought together from the following 2 blog posts

• https://docs.google.com/Doc?id=dgmmft5s_45hr7hmggr
• http://www.manamplified.org/archives/2008/03/notes-on-using-ec2-s3.html

and some tips for the EMR team.