Economist developers are a big, distributed lot. At one point we had 4 teams doing development in 4 different timezones from San Francisco to London. As you might imagine, this presents some extremely interesting challenges, like source code control and maintaining communication channels between teams to reduce the amount of toe stepping and work duplication going on.

Bazaar & Launchpad

The single tool that has had the largest positive impact on code quality and general sanity has been Launchpad. We switched from Subversion to Bazaar / Launchpad in March 2010 and although we didn’t do any official metrics, I would guess we have slashed the defect rate by at least half since doing that.

Bazaar is a distributed source control system that is similar to Git, perhaps marginally easier to use although unfortunately a little slower. Here’s how it works: there’s a trunk branch that contains the latest stable code, and when starting a new piece of work, you branch off trunk and do the work in the branch, until it’s stable enough for review.

At that point, create a merge proposal in Launchpad and any developer on the project can provide a review of the code. The rules we live by are that any of the ~20 developers can review any code, however we recognise that there are a diverse range of skills and encourage both expert reviews from the experts of that specific area, be it domain specific or language specific, as well as reviews from anyone who feels like learning something they may not known much about. Reviewing code is an extremely good way to learn about a part of the system that you may not be familiar with. Yeah, we’re all Drupal developers, but on a site as large as The Economist, it’s really hard to know every nook of the system.

Code reviews

Once code is approved, there is a smaller subset of developers who hold the title of ‘core committer’. These devs have the authority to give final approval for merging the branch to trunk. The core committers’ job is to review the review, and make sure the proper consideration has been given to the overall impact of pushing the branch into trunk.

The impact of this review process is profound. Teams get feedback on everything from coding style to architecture approach, and we’ve seen a great deal of convergence and standardisation in what was (in the old system) essentially just chaos. We can now claim to really follow the same coding standards. Bugs and mistakes are spotted early before getting out into the wild. It’s really effective.

Vendor branches

For each module that we use from Drupal.org, we make a vendor branch in Launchpad and bzr join it into trunk at /sites/all/modules/my_module. That way, we can commit any modifications to the module code directly on the trunk copy (and submit a patch, of course), and when a new version comes out we update the vendor branch and merge it into trunk. Our changes are preserved, and any conflicts indicate that we have made changes that are incompatible with the upstream version.

Releasing

We have multiple teams all doing Scrum, but none of the sprints are aligned. They’re all different durations and can start or end on any day of the week. This means we can’t schedule a release at the end of each team’s sprint as it would be too chaotic. Instead, we have an independent weekly release schedule, that pushes the whole of trunk to live for each release. Remember trunk contains only stable, reviewed and approved code, so it’s always ready to go.

We have a dedicated set of servers in our production data center for staging. It’s an approximate scale model of the actual production cluster and is maintained for testing purposes. It’s useful for testing the actual deployment of the code, or any processes that require access to the live environment, for example any strange interactions with the load balancer.

The first step in a release happens on Friday morning, when we push a copy of trunk to a release branch and tag a release candidate. The release is then deployed to stage and manual testing can happen for a few days with the weekend to spare if we ever discover something catastrophically wrong (thankfully, no weekends have been sacrificed yet).

On Tuesday morning, the release is pushed to live. Bugfix releases are handled in a similar way, except in a compressed time frame.