Last modified 11 years ago Last modified on 09/16/2008 08:35:43 AM

Evaluating Continuous Integration Tools:

Must haves:

  • Display of build status in a form that's easy to see when builds aren't working.
  • "Blame" function to show who the last person to check in a change was

We'd like to have the following builds:

  • Distribution-based build
    • lsstpkg LSSTPipe - done
    • nightly
  • Full SVN build against release
    • build each package against latest released pkgs (order not important)
    • nightly
  • Incremental SVN build against release
    • build only packages with updates against release
    • hourly
  • Full development build
    • build in order all packages from trunk
    • nightly
  • Incremental development build
    • build packages that have changed (in order)
    • build all packages that depend on these packages.

Tinderbox evaluation


  • Integrated with Apache
  • Waterfall Display
  • "cvsblame" function
  • Can be used with SVN


  • No trac functionality
  • No formal "releases", just CVS check-out of the current code
  • Mozilla appears to be evaluating BuildBot as a replacement, so this might be on the tail end of development

BuildBot evaluation:

BuildBot works as a parent bot and a series of child bots. The child bots contact the parent, which hands out work to each of the children. Each child can be individually configured to do work, so that each child can be told to do a different job. Work status is reported back to the parent, which can be queried to the results on a web page.


This picture shows a sample run with a file that was edited to purposely add an error:

Clicking on the user name in the first column reveals the SVN information associated with the changes before this run occurred:

To view the error that occurred, click on the link named "stdio", and you can see the standard output of the attempted compilation:


  • Shows "waterfall" color history display of all bots work
  • Displays history of all SVN check-ins
    • Main display shows when check-in occurred, and who was responsible
    • Displays what was checked in, and SVN commit message
  • Easily customizable; configuration scripts are written in Python
  • Can watch for SVN check-ins to "settle down" before doing a build
  • Developer API to customize build procedures
  • Apache not required
  • Works on multiple platforms; RPMs for CentOS and Redhat, tested under other Linux systems and Mac OS X.
  • Actively being developed, with a Roadmap


Bitten evaluation:

Bitten is a Trac extension that is a multi-platform build monitoring system. The Trac server functions as a master, managing slave machines that are assigned builds for one or more platforms. Platforms are qualified by OS (Linux, Darwin, etc.), version, hardware, and processor (powerpc, athlon, etc.). The master assigns builds to slaves based on both the slaves' capabilities and the platforms the master is configured to monitor.

Communication is always from slave to master; slaves poll the master to ask for instructions. Communication is a variant of BEEP over HTTP(S).


  • Ready access to build history (example)
  • Nice display of unit tests history and code coverage (example)
  • User interface is integrated with Trac
  • Sophisticated parsing of build output
  • Can handle slaves coming and going


  • Doesn't notify anyone when build or tests fail??!!??

Other prominent Continuous Integration products that we are not actively investigating:

  • TinderBox
    • Written in Perl
    • Complicated to build and configure
    • Minimally maintained
    • Mozilla, its progenitor, appears to be considering alternatives
  • Hudson
    • Easy to set up and administer
    • Integration module available for Trac
    • Written in Java
  • Cruise Control