This document describes the setup for building Linux packages (RPMs) of
CellProfiler and its dependencies and publishing them on the web.
SSH is set up to allow logging into the build machines without
paswords, but there have been some problems with this due to a CentOS
6 bug. You should make RSA public and private key files (id_rsa and id_rsa.pub)
in the subdirectory, `ssh_keys`. Then, run `restorecon -R -v /root/.ssh` to make
password-less login work on CentOS 6.
## Environment
Make sure to use Python 2.6 or 2.7, then activate the virtual environment
. venv/bin/activate
## Changing Centos 6 RPM dependencies in CellProfiler
CellProfiler has a spec file stored in its GIT repository:
jenkins/linux/cellprofiler-centos6.spec
Each commit has its own set of dependencies because the
spec file is part of the commit. Dependencies can be
modified and added by editing the "Requires" and
"BuildRequires" sections of the .spec file.
## Files and directories
fabfile.py
: Fabric tasks for building and testing.
htdocs/linux
: This is the directory that is exposed on the web. It contains a subdirectory
: for each Linux distribution. For RPM-based distributions such as CentOS, that
: directory is a yum repository created by createrepo (see below).
ssh_keys
: Contains SSH keys that are installed on the build and test machines by some
: of the fabric tasks.
SPECS
: RPM spec files.
## How to rebuild only CellProfiler
0. Edit SPECS/cellprofiler.spec and update the version, release, and/or
tarname. The tarname should be the git hash. Example:
%define version 2.1.0.Release
%define release 1
%define tarname 0c7fb94
1. Activate the virtualenv.
. venv/bin/activate
2. Make a working directory. This directory will contain RPMs as you build
them and test; you will finally copy them to the website.
$ mkdir foo
$ cd foo
3. Start a clean build machine by going to http://broad.io/imagingcloud,
logging in, and add a vApp based on the template "Cent OS 6 x64" in
the Build Catalog.
4. Wait for the build machine to start and get its IP address,
192.168.195.XXX.
5. Upgrade the operating system and install packages needed to build.
$ fab -H 192.168.195.XXX deploy_build_machine
6. Build the cellprofiler RPM.
$ fab -H 192.168.195.XXX build_cellprofiler_only
7. Start a clean test machine by going to http://broad.io/imagingcloud,
logging in, and add a vApp based on the template "Cent OS 6 x64" in
the Build Catalog.
8. Wait for the test machine to start and get its IP address, 192.168.195.YYY.
9. Upgrade the operating system and install packages needed to test.
$ fab -H 192.168.195.YYY deploy_test_machine
10. Tell the test machine to get the dependencies from the public repository.
$ fab -H 192.168.195.YYY use_public_repo
11. Copy the new RPM to the test machine.
$ fab -H 192.168.195.YYY push
13. Install CellProfiler on the test machine.
$ fab -H 192.168.195.YYY install_cp
(Not 100% sure that this will get the newly built version and not the
version that is in the public repository. Check.)
14. Log in and check that it works to start CellProfiler.
$ ssh -Y johndoe@192.168.195.YYY cellprofiler
15. Add the new RPM files to the public repository.
$ cp *.rpm ../htdocs/linux/centos6/
$ createrepo ../htdocs/linux/centos6/
16. Delete the working directory.
$ cd ..
$ rm -rf foo
## How to build a particular version
1. Create an OAUTH token at github.com (needed to allow the script to
make unlimited calls to the github API). Go to
https://github.com/settings/applications and create a personal access
token with all permissions unchecked.
2. Create and cd into a temporary directory
3. Run fab like this:
$ fab -H 192.168.xxx.xxx build_cellprofiler_version:<git-hash>,<token>
## How to rebuild everything
0. Activate the virtualenv.
. venv/bin/activate
1. Make a working directory. This directory will contain RPMs as you build
them and test; you will finally copy them to the website.
$ mkdir foo
$ cd foo
2. Start a clean build machine by going to http://broad.io/imagingcloud,
logging in, and add a vApp based on the template "Cent OS 6 x64" in
the Build Catalog.
3. Wait for the build machine to start and get its IP address,
192.168.195.XXX.
4. Upgrade the operating system and install packages needed to build.
$ fab -H 192.168.195.XXX deploy_build_machine
5. Copy the contents of the SOURCES directory to the build machine.
$ fab -H 192.168.195.XXX push_sources
6. Build all the RPMs.
$ fab -H 192.168.195.XXX maybe_build_all_rpms
7. If something won't build, modify the .spec file as needed and repeat
step 6 until it works. If you add a file to SOURCES, run step 5 again.
8. Copy all the RPMs from the build machine.
$ fab -H 192.168.195.XXX pull
9. Start a clean test machine by going to http://broad.io/imagingcloud,
logging in, and add a vApp based on the template "Cent OS 6 x64" in
the Build Catalog.
10. Wait for the test machine to start and get its IP address, 192.168.195.YYY.
11. Upgrade the operating system and install packages needed to test.
$ fab -H 192.168.195.YYY deploy_test_machine
12. Copy the RPMs to the test machine.
$ fab -H 192.168.195.YYY push
13. Install CellProfiler on the test machine.
$ fab -H 192.168.195.YYY install_cp
14. Log in and start CellProfiler.
$ ssh -Y johndoe@192.168.195.YYY cellprofiler
15. Add the new RPM files to the public repository.
$ cp *.rpm ../htdocs/linux/centos6/
$ createrepo ../htdocs/linux/centos6/
16. Delete the working directory.
$ cd ..
$ rm -rf foo
## How to build one thing
0. Activate the virtualenv.
. venv/bin/activate
1. Make a working directory. This directory will contain RPMs as you build
them and test; you will finally copy them to the website.
$ mkdir foo
$ cd foo
2. Start a clean build machine by going to http://broad.io/imagingcloud,
logging in, and add a vApp based on the template "Cent OS 6 x64" in
the Build Catalog.
3. Wait for the build machine to start and get its IP address,
192.168.195.XXX.
4. Upgrade the operating system and install packages needed to build.
$ fab -H 192.168.195.XXX deploy_build_machine
5. Copy the contents of the SOURCES directory to the build machine.
$ fab -H 192.168.195.XXX push_sources
6. Push the existing RPMs to the build machine
$ fab -H 192.168.195.XXX push
7. Use public repositories
$ fab -H 192.168.195.xxx use_public_repo
8. Make the single RPM, for instance cellprofiler-foo
$ fab -H 192.168.195.XXX maybe_build_rpm:<package-name>
9. Copy all the RPMs from the build machine.
$ fab -H 192.168.195.XXX pull
## Reindexing a yum repository
After adding or removing RPM files from a yum repository, run createrepo
in order to recreate the index files. For instance:
createrepo htdocs/linux/centos6
## Testing CellProfiler
fab -H 192.168.195.XXX deploy_test_machine
fab -H 192.168.195.XXX install_cp
## How the virtualenv was made
The directory venv is a python virtualenv that contains software needed
to start builds and create repositories. The virtualenv was made as
follows:
1. Create virtualenv
virtualenv --prompt='(linux_repositories)' venv
2. Activate virtualenv
. venv/bin/activate
3. Install createrepo
mkdir venv/src
cd venv/src
wget http://createrepo.baseurl.org/download/createrepo-0.3.6.tar.gz
tar xzf createrepo-0.3.6.tar.gz
cd createrepo-0.3.6
patch < ../../../createrepo-sysconfdir.patch
make prefix=$(dirname $(dirname $(pwd))) install
cd ../../..
# ManuallyrReplace /usr/share with $(dirname $(dirname "$0")) in
# venv/bin/createrepo.
4. Install fabric
pip install fabric