aboucaud / ramp-cloud

RAMP kits virtual image setup

This program aims deploying a RAMP pipeline with a given kit, on a virtual image. Instances with that image can then be fired for each user submission to compute their predictions.

Currently only Amazon EC2 is supported. Microsoft Azure and Google Compute Engine will be next.

Usage

1. install Packer (see dedicated section)

2. set up your cloud credentials (see here)

3. clone this repo

   git clone https://github.com/aboucaud/ramp-cloud.git && cd ramp-cloud

4. select the .json config file in the directory corresponding to the backend provider (aws | azure | gcloud), as well as the hardware you need (cpu | gpu)

5. run Packer and specify the name of the starting kit (used on github.com/ramp_kits) and the backend data directory

   packer build -var ramp_kit_name=<kit_name> -var backend_data_dir=/path/to/data/ config.json

   This will create a virtual image called <kit_name>_backend.

Example: command for creating the backend AWS AMI with pure CPU support for the autism starting kit; where the backend data files are stored in a local directory ./autism/testdata/

packer build -var ramp_kit_name=autism -var backend_data_dir=./autism/testdata/ aws_cpu_setup.json

This will produce an AMI called autism_backend.

Packer

The kit environments on virtual machines (AWS, Azure, GCloud) are build using Packer. To create an image, the first step is to install Packer.

Packer uses a JSON configuration file called template to specify the needs for building an image.

Commands

• check the configuration file

  packer validate config.json

• create the image

  packer build config.json

Template file

Configuration - template - file example

{
  "variables": {
    "aws_access_key": "",
    "aws_secret_key": ""
  },
  "builders": [
    {
      "type": "amazon-ebs",
      "access_key": "{{user `aws_access_key`}}",
      "secret_key": "{{user `aws_secret_key`}}",
      "region": "us-west-2",
      "source_ami": "ami-0bbe6b35405ecebdb",
      "instance_type": "t2.micro",
      "ssh_username": "ubuntu",
      "ami_name": "packer-hello-world"
    }
  ],
  "provisioners": [
    {
     "type": "shell",
     "inline": "echo \"Hello World!\"" 
    }  
  ]
}

The template files consist in three main sections,

• variables: which specifies the user variables and a default value. These variables can be superseeded with environment variables (same name) MY_VAR=ramp or directly in the build command line

  packer build -var my_var=ramp template.json

• builders: which is responsible for creating the machine and building the image. It has information on the platform, the base image, the image name, etc.

• provisioners: the list of actions to perform on the image before saving it. Among those, file upload/download, remote execution of shell commands and shell scripts.

Credentials

Amazon EC2

In order for the Packer scheme to work seemlessly, one needs to set up local credentials in the form of a file ~/.aws/credentials on Linux/macOS ; or C:\Users\USERNAME\.aws\credentials on Windows.

These credentials can be obtained in your Amazon account, see this page for help.

Microsoft Azure

Azure setup

Google Cloud

Google Compute Engine setup

Add ons

Amazon EC2

Once AWS credentials are set locally, a Python command line interface is available to perform a few actions:

• list the EC2 instances by state (default is 'running')

  python aws.py instance list [--state <state>]

• list the registered AMIs

  python aws.py ami list

• given an AMI ID, print the corresponding name

  python aws.py ami name <ami-id>

• deregister an AMI from EC2

  python aws.py ami delete <ami-id> [--dry-run]

This CLI needs Click and the boto3 libraries to work, see requirements.txt.