This codebase tracks work in progress toward a web server that will allow users to apply a working ADAGE model to their own data sets. It should be considered pre-release status. The following instructions detail the steps required for getting a development instance up and running manually. For a guide to automated deployment, see Deployment Steps below.
Note: The following steps assume you have already installed PostgreSQL (>=9.4), NGINX (>=1.10), supervisord (>=3.2), Python (2.7) and Elasticsearch (1.7) on Ubuntu (16.04).
Fork the adage-server repository on Github (see Github's documentation for forking repositories) and then clone that fork you made in the directory of your choice.
cd /<your chosen directory>/
git clone git@github.com:<your github account>/adage-server.git
The file config.py
(in the adage-server/adage/adage folder
) contains all of
the settings that must be edited for deployment. Because this file contains
secrets and deployment-specific information, it is not tracked under source
control. You should copy the file
adage-server/adage/adage/config.py.template
from the repository and use that
as a starting point for your own deployment's config.py
. This puts all of the
secrets and deployment-specific information into a single, easily-controlled
file.
Set the full path of parent directory where adage-server repository was
cloned into in the 'home_dir'
key of the OS_CONFIG
setting.
OS_CONFIG = {
'home_dir': '/<path to directory>/',
}
Set up Database Name, User, Password and Host in databases
settings:
DEV_CONFIG.update({
'databases': {
'default': {
# This example uses psycopg2 for PostgreSQL, but you can use any
# of the Engines supported by Django. For more information, see:
# https://docs.djangoproject.com/en/dev/ref/settings/#databases
'ENGINE': 'django.db.backends.postgresql_psycopg2',
# database name and user cannot have upper case letters
'NAME': '<your_adage_database_name>',
'USER': '<your_adage_db_username>',
'PASSWORD': '<your_db_username_password>',
# Wherever PostgreSQL is being hosted,
# usually localhost for development
'HOST': 'localhost',
# Port where it is being hosted from,
# usually 5432
'PORT': '5432',
}
},
})
Other settings in DEV_CONFIG
may be left alone unless you are using the
fabric
scripts as described in Deployment Steps.
If you intend to use features from Tribe in your installation of adage-server
,
you will need to follow the instructions for setting up tribe-client
found
on its PyPI page: https://pypi.python.org/pypi/tribe-client and fill in the
corresponding TRIBE_*
variables in config.py
.
Change the last line of the file to read:
CONFIG = DEV_CONFIG
# Switch to postgres user, enter the superuser password when you
# are prompted
sudo su - postgres
# Create adage-server database specified in previous step
createdb <your_adage_database_name>
# Create adage-server database user specified in previous step
createuser -P <your_adage_db_username>
# This prompts you to enter the password for new role/user (also
# the one specified in config.py file in previous step)
# Enter psql interface:
psql
# Give all privileges of this newly created database to the
# newly created user
GRANT ALL PRIVILEGES ON DATABASE <your_adage_database_name> TO <your_adage_db_username>;
# Also give this user permissions to create databases - this is needed to be
# able to run the Django test suite (since a test database is created).
ALTER USER <your_adage_db_username> CREATEDB;
# Start a Python virtual environment in the adage-server root
# or wherever you keep your Python virtual environments
virtualenv <virtual envs location>/adage
source <virtual envs location>/adage/bin/activate
# Install Python requirements
cd adage/
pip install -r requirements.txt
python manage.py migrate
Download the get_pseudo_sdrf.py
and gen_spreadsheets.py
files from
this repository,
and put them in whichever parent directory the adage-server repository has been
cloned into.
Django will use the database you configured earlier to build tables to support
its models (during the migrate
step, above), but the tables will remain empty
until you run the management commands to import data into the models. This
repository contains several data files from our work with Pseudomonas
aeruginosa that we use for "bootstrapping" our database. The commands to load
the data are scripted in our fabric
deployment scripts. If you choose, you
may run those commands at this point using the following steps:
-
Install
fabric
, the tool we use for scripting deployment steps:> pip install fabric
-
Ensure your virtual environment path is set in
config.py
. If the path to the virtual environment you created above matches thevirt_env
specified inconfig.py
, you are all set (by default, theDEV_CONFIG
will inherit that field fromAWS_CONFIG
). If not, you will need to add this setting to yourDEV_CONFIG
because the command in the next step makes use of it.The command below also uses the
django_dir
setting. If you have specified yourhome_dir
and set up your clone of the repository in a directory namedadage_server
within thehome_dir
, then this will match the default configuration. If not, you will also need to specify thedjango_dir
explicitly in yourDEV_CONFIG
. -
Run the following commands from your local clone of the adage-server repository:
> cd <your adage-server directory> > fab adage_server.init_instance:<user>@<hostname>
Fabric is configured to run commands on a remote server. Specify a host name or IP address for the
hostname
you are setting up (you can just specifylocalhost
if you are configuring the local account and you have an ssh server running) and specify theuser
with the clone of the repository.Fabric will execute commands to populate the database and rebuild the search index automatically.
This is a lengthy process that will take over an hour to retrieve and load data
to initialize all models in the adage-server
with data from our Pseudomonas
aeruginosa work. If this is not of interest, refer to the commands in
import_data_and_index()
from fabfile/adage_server.py
in this repository for
an idea of how to tailor this step for your own use. The python management
commands in that script may be run manually with your own data files.
Documentation for using each command is available by typing:
> python manage.py help <command>
This step downloads and pickles all the public Tribe gene sets for every
organism that has been loaded into the database. These pickled files are
necessary for doing gene set enrichment analyses. For more information, see
the tribe-client
documentation found on
its PyPI page.
python manage.py tribe_client_pickle_public_genesets
cd interface/
# Download newest version of Node.js. Also, the command
# "curl -sL https://deb.nodesource.com/setup_4.x | sudo -E bash -"
# is needed before the "sudo apt-get install -y nodejs" if you are
# running a Ubuntu version older than 16.04.
sudo apt-get install -y nodejs
# Install needed interface packages
sudo npm -g install grunt-cli karma-cli bower
npm install
bower install
# Run Grunt to build the interface
grunt
-
Follow the steps to edit settings in config.py as described above.
-
Fork the deployment repository and then clone that fork to a directory alongside your
adage-server
repository.> cd /<your chosen directory>/ > git clone git@github.com:<your github account>/adage-deploy.git
-
Install
fabric
, the tool we use for scripting deployment steps:> pip install fabric
-
If deploying to AWS, you also need to install
boto3
and its requirements:> pip install boto3
-
To perform an AWS deployment, ensure your RDS instance is online, the
AWS_DEPLOY
section ofconfig.py
is properly completed, and run the followingfabric
command:> fab deploy_aws
This will spin up a new EC2 instance using the AWS credentials found in
config.py
, deploy the latestadage-server
code from GitHub and configure all required services. Configuring DNS to direct your domain to the new server must be done manually. (We have DNS pointing to an Elastic IP address and simply re-associate to the new server when deployment succeeds. -
To perform deployment to a new development server, run the following
fabric
command:> fab deploy_dev
This will execute the same deployment steps as run for AWS deployment, but skips the step that spins up an EC2 instance and makes a configuration tweak that allows
nginx
to respond to requests for any hostname or IP address. This method assumes you have a fresh installation of Ubuntu 16.04 and that you have configured theDEV_CONFIG
section ofconfig.py
with credentials for a user withsudo
privileges that can be used to create the requisite services and a user account with the minimum privileges required to host the deployedadage-server
code.