When making a new release we often ran into the problem that files were missing from the source archive. The common error was that the compilation failed due some missing header files.

The second problem came a bit later. As part of the growth of OpenBSC/Osmocom we took code from OpenBSC and moved it into a library called libosmocore to be used by other applications. In the beginning our API and ABI of this new library was not very stable. One thing that could easily happen is that we updated the API, migrated OpenBSC to use the new API but forgot to update one of the more minor projects, e.g. our TETRA decoder. 

The solution is quite simple. The GNU Automake buildsystem already provides a solution to the first problem. One simply needs to call make distcheck. This will create a new tarball and then build it. Ideally all developers use make distcheck before pushing a change into our repository but in reality it takes too much to do this and one easily forgets this step.

Luckily CPU time is getting more and more affordable. This means that we can have a system that will run make distcheck after each commit. To address the second part of the problem we can rebuild all users of a specific library, and do this recursively.

The buzzword for this is Continuous Integration and the system of our choice is Jenkins (formerly known as Hudson). Jenkins has the concept of a Job and a Node. A Job can be building a certain project, e.g. building libosmocore. A Node is a physical system with a specific compiler. A Job can instruct Jenkins to monitor our git repositories and then schedule the job to be build.

In our case we have nodes for FreeBSD/AMD64, Debian6.0/i386 and mingw/i386. All our projects are multi-configuration projects. For some of our Jobs we use it to build the software on FreeBSD, Debian and mingw for others only on Debian. Another useful feature is the matrix build. This way one job can build several different configurations, e.g. debug and release.

Jenkins allows us to have dependencies between Jobs and we are using this to rebuild the users of a library after a change, e.g. build libosmo-abis after libosmocore.

The build-status can be reported by EMail, irc but I generally use the RSS feed feature to find out about broken builds. This way I will be made aware of build breakages and can escalate it by talking to the developer that has caused the breakage.

The installation of Jenkins makes sure that the tarballs built with make dist contains everything needed to build the software package and we have no silent build breakages in less active sub-projects. A nice side-effect is that we have less Emails from users due build breakages. Setting up Jenkins is easy and everyone building software should have Jenkins or a similar tool.

We could have more build nodes for more Linux distributions and versions. This mainly depends on volunteers donating CPU time and maintaining the Jenkins Node. Jenkins offers a variety of plugins and it appears to be easy to write new plugins. We could have plugins that monitor and plot the binary size of our libraries, check for ABI breakages, etc.