=begin rdoc Neptune: A Domain Specific Language for Deploying HPC Software on Cloud Platforms Neptune provides programmers with a simple interface by which they can deploy MPI, X10, MapReduce, UPC, and Erlang jobs to without needing to know the particulars of the underlying cloud platform. You only need to give Neptune your code, tell it how many machines to run on and where to put the output: Neptune handles everything else. No more writing configuration files, having to start up NFS on all your machines, yada yada yada. Neptune works together with supported cloud platforms (currently AppScale is recommended) and can deploy over anything AppScale can - Xen or KVM virtual machines as well as Eucalyptus and Amazon EC2. There's nothing virtualization specific in there, so in theory any machine installed with the AppScale software should work fine. Although Neptune is designed to automate deploying HPC jobs, it also can be used to deploy other types of software. For example, Neptune has support for user-specified scaling of the underlying cloud platform: users can write Neptune jobs that manually add load balancers, application servers, or database servers to a running AppScale deployment. Additionally, as of version 0.0.2, remote compiling can also be performed: just give Neptune the path to the directory you want to compile and be sure to include a Makefile in it! Neptune will run 'make' on it (you can specify which target to make as well) and return to you a folder containing the standard out and standard error of the make command. By default, Neptune jobs store their outputs in the underlying database that AppScale is running over. As of Neptune 0.0.5, job outputs can also be stored in Amazon S3, Eucalyptus Walrus, and Google Storage. Sample Neptune job scripts can be found in samples. Test cases can be found in the test folder, with the standard naming convention - ts_neptune is the test suite runner, with tc_* containing test cases for each type of job that Neptune offers. Before running ts_neptune, you should export the environment variable APPSCALE_HEAD_NODE, which should be set to the IP address of the AppScale machine that runs the Shadow daemon (a.k.a. the Master AppController). Running generate_coverage.sh in the top-level directory will run rcov and generate the coverage reports automatically - currently this process takes about an hour. Developed by Chris Bunch as part of the AppScale project. See LICENSE for the specifics of the New BSD License by which Neptune is released. Check us out on the web: http://neptune-lang.org http://code.google.com/p/appscale http://appscale.cs.ucsb.edu Contributors welcome! We would love to add support for other cloud platforms and test out Neptune more on non-virtualized deployments, as well as adding capabilities for other types of computation. We would also like to refactor Neptune to use symbols instead of instance variables for running jobs: this will likely appear in a future release as well. An academic paper describing Neptune is in the works - check in for a link to that as it becomes available. Version History: April 8, 2011 - 0.0.8 released, fixing MapReduce support for both regular Hadoop and Hadoop Streaming. Also increased code coverage to cover a number of failure scenarios. April 2, 2011 - 0.0.7 released, adding automatic test suite and many bug fixes for all scenarios. rcov can also be used to generate test coverage information: current coverage stats can be found in coverage directory. MapReduce broken at the moment - will fix in next release March 28, 2011 - 0.0.6 released, adding support for input jobs, so users can place data in the datastore without having to run any computation March 18, 2011 - 0.0.5 released, adding support for storage outside of AppScale to be used. Tested and working with Amazon S3 and Google Storage February 10, 2011 - 0.0.4 released, adding UPC and Erlang support, and restructuring syntax to pass in hashes to method calls instead of passing in blocks February 4, 2011 - 0.0.3 released, allowing users to use Neptune properly as a gem within Ruby code February 4, 2011 - 0.0.2 released, adding support for remote compiling January 27, 2011 - 0.0.1 released, with initial support for MPI, X10, and MapReduce. =end