keystone-example
An example skeleton building applications on top of KeystoneML.
To Run This Example:
Launch a KeystoneML cluster using the provided scripts according to these instructions.
Once the cluster is up, ssh onto the master node and execute these commands:
# Build keystone-example
cd /mnt
git clone https://github.com/amplab/keystone-example.git
cd keystone-example
sbt/sbt assembly
~/spark-ec2/copy-dir target/
# Get the data
wget http://qwone.com/~jason/20Newsgroups/20news-bydate.tar.gz
tar -xvzf 20news-bydate.tar.gz
# Copy to HDFS
/root/ephemeral-hdfs/bin/hadoop fs -copyFromLocal 20news-bydate-train/ /data/
/root/ephemeral-hdfs/bin/hadoop fs -copyFromLocal 20news-bydate-test/ /data/
# Run the pipeline
export SPARK_HOME=/root/spark
export KEYSTONE_HOME=/root/keystone
KEYSTONE_MEM=4g ./bin/run-pipeline.sh \
pipelines.ExamplePipeline \
--trainLocation /data/20news-bydate-train \
--testLocation /data/20news-bydate-test