adage-server

This codebase tracks work in progress toward a web server that will allow users to apply a working ADAGE model to their own data sets. It should be considered pre-release status. The following instructions detail the steps required for getting a development instance up and running manually. For a guide to automated deployment, see Deployment Steps below.

Get a working instance of the adage-server running

Note: The following steps assume you have already installed PostgreSQL (>=9.4), NGINX (>=1.10), supervisord (>=3.2), Python (2.7) and Elasticsearch (1.7) on Ubuntu (16.04).

Fork and clone the adage-server repository

Fork the adage-server repository on Github (see Github's documentation for forking repositories) and then clone that fork you made in the directory of your choice.

cd /<your chosen directory>/
git clone git@github.com:<your github account>/adage-server.git

Edit settings in config.py file

The file config.py (in the adage-server/adage/adage folder) contains all of the settings that must be edited for deployment. Because this file contains secrets and deployment-specific information, it is not tracked under source control. You should copy the file adage-server/adage/adage/config.py.template from the repository and use that as a starting point for your own deployment's config.py. This puts all of the secrets and deployment-specific information into a single, easily-controlled file.

Set the full path of parent directory where adage-server repository was cloned into in the 'home_dir' key of the OS_CONFIG setting.

OS_CONFIG = {
    'home_dir':  '/<path to directory>/',
}

Set up Database Name, User, Password and Host in databases settings:

DEV_CONFIG.update({
    'databases': {
        'default': {

            # This example uses psycopg2 for PostgreSQL, but you can use any
            # of the Engines supported by Django. For more information, see:
            # https://docs.djangoproject.com/en/dev/ref/settings/#databases

            'ENGINE': 'django.db.backends.postgresql_psycopg2',

            # database name and user cannot have upper case letters
            'NAME': '<your_adage_database_name>',
            'USER': '<your_adage_db_username>',
            'PASSWORD': '<your_db_username_password>',

            # Wherever PostgreSQL is being hosted,
            # usually localhost for development
            'HOST': 'localhost',

            # Port where it is being hosted from,
            # usually 5432
            'PORT': '5432',
        }
    },
})

Other settings in DEV_CONFIG may be left alone unless you are using the fabric scripts as described in Deployment Steps.

If you intend to use features from Tribe in your installation of adage-server, you will need to follow the instructions for setting up tribe-client found on its PyPI page: https://pypi.python.org/pypi/tribe-client and fill in the corresponding TRIBE_* variables in config.py.

Change the last line of the file to read:

CONFIG = DEV_CONFIG

Set up the database name, user, and password on PostgreSQL

# Switch to postgres user, enter the superuser password when you
# are prompted
sudo su - postgres

# Create adage-server database specified in previous step
createdb <your_adage_database_name>

# Create adage-server database user specified in previous step
createuser -P <your_adage_db_username>
# This prompts you to enter the password for new role/user (also
# the one specified in config.py file in previous step)

# Enter psql interface:
psql

# Give all privileges of this newly created database to the
# newly created user
GRANT ALL PRIVILEGES ON DATABASE <your_adage_database_name> TO <your_adage_db_username>;

# Also give this user permissions to create databases - this is needed to be
# able to run the Django test suite (since a test database is created).
ALTER USER <your_adage_db_username> CREATEDB;

Install Python dependencies

# Start a Python virtual environment in the adage-server root
# or wherever you keep your Python virtual environments
virtualenv <virtual envs location>/adage
source <virtual envs location>/adage/bin/activate

# Install Python requirements
cd adage/
pip install -r requirements.txt

Run Django migration of database tables

python manage.py migrate

Download other necessary files

Download the get_pseudo_sdrf.py and gen_spreadsheets.py files from this repository, and put them in whichever parent directory the adage-server repository has been cloned into.

Populate the database

Django will use the database you configured earlier to build tables to support its models (during the migrate step, above), but the tables will remain empty until you run the management commands to import data into the models. This repository contains several data files from our work with Pseudomonas aeruginosa that we use for "bootstrapping" our database. The commands to load the data are scripted in our fabric deployment scripts. If you choose, you may run those commands at this point using the following steps:

Install fabric, the tool we use for scripting deployment steps:
```
> pip install fabric
```
Ensure your virtual environment path is set in config.py. If the path to the virtual environment you created above matches the virt_env specified in config.py, you are all set (by default, the DEV_CONFIG will inherit that field from AWS_CONFIG). If not, you will need to add this setting to your DEV_CONFIG because the command in the next step makes use of it.

The command below also uses the django_dir setting. If you have specified your home_dir and set up your clone of the repository in a directory named adage_server within the home_dir, then this will match the default configuration. If not, you will also need to specify the django_dir explicitly in your DEV_CONFIG.
Run the following commands from your local clone of the adage-server repository:
```
> cd <your adage-server directory>
> fab adage_server.init_instance:<user>@<hostname>
```
Fabric is configured to run commands on a remote server. Specify a host name or IP address for the hostname you are setting up (you can just specify localhost if you are configuring the local account and you have an ssh server running) and specify the user with the clone of the repository.

Fabric will execute commands to populate the database and rebuild the search index automatically.

This is a lengthy process that will take over an hour to retrieve and load data to initialize all models in the adage-server with data from our Pseudomonas aeruginosa work. If this is not of interest, refer to the commands in import_data_and_index() from fabfile/adage_server.py in this repository for an idea of how to tailor this step for your own use. The python management commands in that script may be run manually with your own data files. Documentation for using each command is available by typing:

> python manage.py help <command>

Pickle Tribe gene sets

This step downloads and pickles all the public Tribe gene sets for every organism that has been loaded into the database. These pickled files are necessary for doing gene set enrichment analyses. For more information, see the tribe-client documentation found on its PyPI page.

python manage.py tribe_client_pickle_public_genesets

Build adage-server web interface

cd interface/

# Download newest version of Node.js. Also, the command
# "curl -sL https://deb.nodesource.com/setup_4.x | sudo -E bash -"
# is needed before the "sudo apt-get install -y nodejs" if you are
# running a Ubuntu version older than 16.04.
sudo apt-get install -y nodejs

# Install needed interface packages
sudo npm -g install grunt-cli karma-cli bower
npm install
bower install

# Run Grunt to build the interface
grunt

Deployment Steps

Follow the steps to edit settings in config.py as described above.
Fork the deployment repository and then clone that fork to a directory alongside your adage-server repository.
```
> cd /<your chosen directory>/
> git clone git@github.com:<your github account>/adage-deploy.git
```
Install fabric, the tool we use for scripting deployment steps:
```
> pip install fabric
```
If deploying to AWS, you also need to install boto3 and its requirements:
```
> pip install boto3
```
To perform an AWS deployment, ensure your RDS instance is online, the AWS_DEPLOY section of config.py is properly completed, and run the following fabric command:
```
> fab deploy_aws
```
This will spin up a new EC2 instance using the AWS credentials found in config.py, deploy the latest adage-server code from GitHub and configure all required services. Configuring DNS to direct your domain to the new server must be done manually. (We have DNS pointing to an Elastic IP address and simply re-associate to the new server when deployment succeeds.
To perform deployment to a new development server, run the following fabric command:
```
> fab deploy_dev
```
This will execute the same deployment steps as run for AWS deployment, but skips the step that spins up an EC2 instance and makes a configuration tweak that allows nginx to respond to requests for any hostname or IP address. This method assumes you have a fresh installation of Ubuntu 16.04 and that you have configured the DEV_CONFIG section of config.py with credentials for a user with sudo privileges that can be used to create the requisite services and a user account with the minimum privileges required to host the deployed adage-server code.

jaclyn-taroni / adage-server