Buildbot Sample Configuration

This document describes how to do continuous integration of webapps with Buildbot. It contains configuration files for Buildbot, Git, Apache and Nginx. It's the follow up of this story.

1. Scenario

One Django pluggable application.
One Django project that uses the previous application.
Buildbot has to run tests for both, the app and the project.
Buildbot will build the app when changes hit app's repository.
Buildbot will build the project when changes hit project's repository and right after building the app.
Project's repository will be hosted in an in-house server.
App's repository will be hosted in GitHub.
App and project have to be build under supported versions of Python/Django.
Build results have to be available through a web interface.

2. Solution

Buildbot's master and slave will live in the same machine.
There will be 3 Python virtualenv to support Django v1.4/v1.5 under Python v2.7, and Django v1.5 under Python v3.2.
Buildbot will have 3 slaves one per virtualenv.
Virtualenvs will get active at OS startup time, before launching slaves.
Project's repository in the in-house server will get a new post-receive hook script that will notify Buildbot on changes.
Buildbot will accept HTTP POST requests from GitHub.
App's GitHub repository will get a new WebHook URL pointing to Buildbot's web interface.
Apache or Nginx will handle requests to both Buildbot web interface.

3. Setup

The following setup has been made in a fresh KVM virtual machine running Ubuntu Server 12.04 with the following extra packages installed:

root@server:~# apt-get install apache2 python-pip python-virtualenv python3 git

The Setup consists of the following steps:

Setup the in-house Git repository for the Django project
Setup the GitHub Git repository for the Django app
Install Buildbot
Configure Buildbot
Web server setup
Run at system startup

3.1 Setup the in-house Git repository for the Django project

Django-sample-project it's an implementation of the official Django tutorial. It represents the Django project for the sample configuration.

The following steps create the private in-house Git repository for the project. Create a git user and group, and add your username to the git group:

root@server:~# useradd -m -s /bin/bash git
root@server:~# usermod -G git youruser

Clone a bare copy of django-sample-project to start off the in-house repository:

root@server:~# su - git
git@server:~$ git clone --bare git://github.com/danirus/django-sample-project.git

Copy the post-receive file to /home/git/django-sample-project.git/hooks/. The post-receive hook runs a script that will notify buildbot when changes hit the project's repository.

If your Buildbot's master lives in a different host:port add the --master ipaddress:port option to the hook along with the --repository option.

3.2 Setup the Github repository for the Django app

This step requires the server to be reacheable from the outside world as GitHub will post a HTTP request to Buildbot. If it's not (because you are playing around with a VM, just do the same with the app as what you've done with the project).

Fork django-sample-app in your own GitHub account. Then go to the repository settings, click on Service Hooks, and WebHook URLS. Add the URL of your Buildbot master with the path to the GitHub hook:

http://buildbot.myservice.info/change_hook/github

Buildbot's path for GitHub defaults to change_hook/github.

3.3 Install Buildbot

Use this PPA to install Buildbot v0.8.7 in Ubuntu 12.04. The release provided by default with Ubuntu 12.04 (Buildbot v0.8.5) contains a bug that produces HTTP 500 Server Errors when receiving notifications from GitHub. Version 0.8.7 solves that bug and many others.

After installing the PPA create the user buildbot and install the package:

root@server:~# useradd -m -s /bin/bash buildbot
root@server:~# apt-get install buildbot

root@server:~# su - buildbot
buildbot@server:~$ buildbot create-master master

Each slave will run a different combination of Python and Django:

Python 2.7 and Django 1.4
Python 2.7 and Django 1.5
Python 3.2 and Django 1.5

Create the virtualenvs:

buildbot@server:~$ virtualenv slaves/py27-dj14
buildbot@server:~$ source slaves/py27-dj14/bin/activate
(py27-dj14)buildbot@server:~$ pip install "Django==1.4.5"
(py27-dj14)buildbot@server:~$ deactivate

buildbot@server:~$ virtualenv slaves/py27-dj15
buildbot@server:~$ source slaves/py27-dj15/bin/activate
(py27-dj15)buildbot@server:~$ pip install "Django==1.5.1"
(py27-dj15)buildbot@server:~$ deactivate

buildbot@server:~$ virtualenv -p python3 slaves/py32-dj15
buildbot@server:~$ source slaves/py32-dj15/bin/activate
(py32-dj15)buildbot@server:~$ pip install "Django==1.5.1"
(py32-dj15)buildbot@server:~$ deactivate

Each slave runs inside their own virtualenv. The syntax of the buildslave command is as follows:

buildslave create-slave <basedir> <master-addr/port> <name> <password>

Create the slaves:

buildbot@server:~$ buildslave create-slave slaves/py27-dj14/slave localhost:9989 py27dj14 pass
buildbot@server:~$ buildslave create-slave slaves/py27-dj15/slave localhost:9989 py27dj15 pass
buildbot@server:~$ buildslave create-slave slaves/py32-dj15/slave localhost:9989 py32dj15 pass

3.4 Configure Buildbot in 3 steps

I approach the configuration in three steps to make a kind of a tutorial. Skip it if you already have a good understanding of Buildbot concepts. Then simply use the file master.cfg.project+app+connected. Otherwise go through the 3 steps.

In the first step Buildbot will build only the project. Then I'll add builds for the app. And at the end I'll add steps to trigger the build of the project after building the app. Read Continuous Integration of webapps with Buildbot to get a more wording introduction to this example.

Note: In the following steps you will run Buidlbot from the command line. Use the script provided in the following up sections to integrate it as part of the server startup process.

3.4.1 Setup only the web project

Copy the master.cfg.project file to your /home/buildbot/master/ directory and rename it to master.cfg. Then start the master and the three slaves:

buildbot@@server:~$ buildbot start master

The command will output a bunch of log entries and finish with a line saying The buildmaster appears to have (re)started correctly.

You will get enough information as to guess what's the problem if the command goes wrong. In such a case edit the master.cfg file, fix the bug and restart the service with:

buildbot@server:~$ buildbot restart master

Visit the URL exposed by Buildbot: http://buildbot-server-ip:8010. Visit the waterfall and see that the 3 builders are offline. Start the slaves:

buildbot@server:~$ source slaves/py27-dj14/bin/activate
(py27-dj14)buildbot@server:~$ buildslave start slaves/py27-dj14/slave
[...some log entries...]
The buildslave appears to have (re)started correctly
(py27-dj14)buildbot@server:~$ deactivate

buildbot@server:~$ source slaves/py27-dj15/bin/activate
(py27-dj15)buildbot@server:~$ buildslave start slaves/py27-dj15/slave
[...some log entries...]
The buildslave appears to have (re)started correctly
(py27-dj15)buildbot@server:~$ deactivate

buildbot@server:~$ source slaves/py32-dj15/bin/activate
(py32-dj15)buildbot@server:~$ buildslave start slaves/py32-dj15/slave
[...some log entries...]
The buildslave appears to have (re)started correctly
(py32-dj15)buildbot@server:~$ deactivate

Builders should be idle now in the web interface. Click on any of the builders and click on Force to immediately run a build.

Buildbot's master components (change source, schedulers, filters, build steps, builder factories and builders) required for this step are represented in the following figure:

3.4.2 Add the setup for the web app

Buildbot allows you to build more than one software project. You can run as many as you want. The master configuration file for this step adds the Django app to the building process. Copy the master.cfg.project+app file to your /home/buildbot/master/ directory and rename it to master.cfg.

Edit the file and replace the web app URL with your own. Almost at the top of the file you will find the repos dictionary. Just edit the URL to satisfy the location you use:

repos = {
    'webproject': {
        'url': '/home/git/django-sample-project.git',
        'branch': 'master'
    },
    'webapp': {
        'url': 'https://github.com/<yourGitHugUser>/django-sample-app.git',
        #'url': '/home/git/django-sample-app.git',
        'branch': 'master'
    },
}

To customise the SMTP settings and receive email notification on failed builds adapt the smtp_kwargs and remove the hash from the 3 lines defining the MailNotifier.

Then restart the master (no need to restart the slaves):

buildbot@@server:~$ buildbot restart master

The configuration doesn't change much, now:

The status allows notification from GitHub.
There's a new scheduler for the webapp.
Two filters to see whether source code changes come from the project or the app.
Build steps to build the app.
A build factory to putgroup together those new build steps.
Three new builders for the app to say what steps will build the app and in which slave.

An image is worth a thousand words:

3.4.3 Build the project after building the app

Are you looking for a Continuous Integration tool to build a project based on build results of other project? Buildbot does it hands down and Plugins-free.

The master configuration file for this step runs project builds once their app counterparts have run successfully. What does that mean?

A web app source code change lands in Buildbot.
Buildbot triggers the three app builders.
The last step of each builder will trigger the project builder that runs under the very same conditions (Python+Django combination) only if the app did build successfully.

Copy the master.cfg.project+app+connected file to your /home/buildbot/master/ directory and rename it to master.cfg.

The new configuration adds:

Three triggerable schedulers that will be called from the app builders.
Three new special build steps called Trigger, that will call the triggerable schedulers.
One specific BuildFactory for each app Builder (rather than one for the three), as to add the Trigger step at the end of each BuildFactory. This way each app build triggers the corresponding project builder.

Again, an image's better to illustrates the new scenario:

3.5 Do continuous integration

The app and the project include a couple of test cases to play around with Continuos Integration.

The app comes with a function called do_something (module utils.py) that is used in the function do_otherthing, in the utils.py module of the project. Both, app and project, include tests for their respective functions.

Change do_something to make it return a string rather than an integer and adapt the test case in sample_app/tests/utils_tests.py to make the app pass their own tests (run them manually first with python setup test.py). Then push the changes to the repository and see that the project fails to build. Be sure that your project uses your app's repository in the requirements file rather than mine, otherwise it won't fail.

3.6 Web server setup

Buildbot's web interface can be publicly reacheable through Apache, Nginx or any other web server with proxy capabilities. Checkout the simple sample virtual host configuration files provided for both Apache and Nginx:

The configuration makes the web server act as a proxy to pass all incoming requests to buildbot. It also setup restricted access to the path /change_hook/github/ through which GitHub will post source code changes to Buildbot. Be sure that the list of IP addresses included are the same GitHub enables after setting up your WebHook.

3.7 Run at system startup

Use the following files with the init scripts provided by the Debian/Ubuntu package for Buildbot. They will make Buildbot run the slaves in their appropriate virtualenv:

Copy etc.default.buildmaster to /etc/default/buildmaster
Copy etc.default.buildslave to /etc/default/buildslave
Copy activate_venv.sh to /home/buildbot/slaves

Doing so there won't be any conflict with Buildbot packages in case of updates from Debian/Ubuntu.

danirus / buildbot-sample-conf