Hacking B2G: Building with Distcc

Posted at March 5, 2014

I just landed a patch to fix Bug 970567 that makes it possible to do full B2G builds using distcc. It is a small change that forces the make files to use absolute paths when running the toolchain executables. With this change, it is possible to set up a distcc cluster that is capable of building both host and target files.

My cluster consists of my laptop, a quad-core Retina Macbook Pro running Linux, and an 8-core Xeon workstation also running Linux. With distcc, my build times have reduced rougly 50%. I have also switched to using a tmpfs ramdisk for my ccache directory to further speed up builds. Below is a graph illustrating the build speedup gained by using distcc and a ramdisk ccache.

distcc build times

Set Up Distcc

The first step to building your own distcc cluster to get get mulitple computers up and running on your network. I personally believe that linux machines are the easiest to set up for this. You then need to install distcc, ccache and the toolchain for your target. I use Xubuntu on all of my machines. The command to install distcc and ccache is shown below.

$ sudo apt-get install distcc ccache

The next step is to set up the target toolchain on all of your machines. One important thing to remember is that you must replicate the exact directory tree on all of your machines for the distcc build to work properly. For instance, I have clone the B2G repo to /opt/B2G on my laptop. After running the ./config.sh script, the toolchain for the target is found in the /opt/B2G/prebuilt directory. To make distcc work, I created /opt/B2G on my workstation and then copied the contents of the /opt/B2G/prebuilt directory from my laptop to the workstation. In the future, I hope to have a helper script for automating the set up of the toolchain on multiple machines. On your main workstation, run the following commands. This will take care of creating the correct directory tree and copying the toolchain over.

$ cd 
$ ssh builder1 mkdir -p pwd
$ scp -r prebuilt builder1:pwd/prebuilt

After setting up your build cluster, you will need to configure your main workstation so that distcc knows about your build cluster. This is done with a distcc hosts file. On my laptop, distcc looks in my $HOME/.distc for a file named hosts. The contents of my host file looks like the following.

builder1/16,lzo
localhost/8,lzo

This tells distcc to use builder1 first, localhost second. It also specifies the maximum number of jobs it should send to each host (e.g. 16 for builder1 and 8 for localhost) as well as turning on compression for sending data over the network. Once you have this in place you can run distcc and have it display its list of build hosts.

$ distcc --show-hosts
builder1/16,lzo
localhost/8,lzo

Set Up B2G Build

The existing B2G build system uses ccache by default to speed up rebuilds by caching built files to be re-used in subsequent rebuilds. Using ccache means we have to stick with the plain distcc build process because ccache is not compatible with the newer distcc-pump mode. The ccache man page specifies the use of the CCACHE_PREFIX as the right way to integrate with distcc. The correct way to properly set up the environment variables for building B2G is to have a custom .userconfig file in your B2G root. I describe my full .userconfig file in another blog post. A minimal .userconfig for using distcc would look like the following.

To execute a build using distcc you run the following command.

$ USE_DISTCC=1 ./build.sh -j

By only specifying the -j switch without a number, distcc will use all of the available parallelism in the distcc cluster when needed. So in my case, it will send 16 jobs to builder1 and 8 to localhost at once. If you want a nice GUI for watching your parallel builds, you can install the distccmon application.

$ sudo apt-get install distccmon-gnome

If you run distccmon while running a distcc build you should see something like the screenshot below.

screenshot of distccmon