We have a fun concept here that we call a “human test instance”. Anyone can create an entire copy of Economist.com, running in the cloud, on a subdomain of their choice, on a branch of their choice, with just a few clicks in Hudson. “Human testing” implies that the instance is used by a human to do manual testing, however, we use a human test instance to run automated Selenium tests, and each one also effectively tests the process of running the update functions every day.

To accomplish this we use two cloud servers, one running MySQL and another with the rest of the stack (Varnish, Memcache, Apache, PHP). Hudson has them both setup as remote slaves.

As a user, you have to create two jobs to setup a human test instance: a database import job and the actual human test job that points at the branch you’d like to test.

Let’s say we want to create a human test of the trunk branch, on a database named trunk_daily. The first step is to create a database job called trunk_daily by cloning an existing database job, and build it, letting Hudson perform the magic of creating a new copy of production using the name of the job for the database name. It works like this:

1. Connect to the MySQL server slave and download the latest gzipped dump of the production database, only if it hasn’t already been downloaded by another database job.
2. Create a temporary database, import the data. This takes just over 2 hours.
3. Drop and recreate the target database, in this example it’s trunk_daily.
4. Move all the tables from the temp database to the target. This is quick, so the volatile period is actually very short.

We now have a complete copy of the site’s data, ready to go. The next step is to set the job to build on a schedule, for example every night, so that in the morning when we get in, the human test instance will be running against the latest production data.

The second job we create is the human test job, and this is named after the subdomain that we want the new site on. Let’s name it trunk, so that once it’s built, the new environment will be available on http://trunk.humantest.economist.com/.

The job takes a string parameter called HUMAN_TEST_DB_NAME, and we’ll put trunk_daily in that, which is the name of the database we created above. It also wants a path to the Bazaar branch that it will use to power the site. We give it a link to our trunk branch in Launchpad.

Now the magic. When the job builds, it:

1. Checks out a local copy of the given branch in a workspace folder in the Apache webroot, for example, /var/www/html/trunk/
2. Copies our Economist default settings.php into an appropriately named sites folder, called /sites/trunk.humantest.economist.com/settings.php in this case.
3. Modifies the settings to point to the database that we specified.
4. Modifies the memcache prefix so that this human test instance has it’s own namespace in the shared instance.
5. Use drush to run all outstanding updates, and clear the cache.
6. Navigate to the human test in your browser, and there is a tiny bit more magic that happens. The first is a wildcard DNS entry at *.humantest.economist.com that points at the human test server. The second is an Apache configuration that dynamically changes the webroot depending on the incoming request’s server name:

<VirtualHost _default_:8080>
    RewriteEngine on
    ServerAlias *.humantest.economist.com
    UseCanonicalName Off
    VirtualDocumentRoot /var/www/html/workspace/%1/drupal
</VirtualHost>

Going to anythingatall.humantest.economist.com will have Apache searching for index.php at /var/www/html/workspace/anythingatall/drupal/index.php.

The human test Hudson jobs are configured to ping Launchpad and check for any new commits on the branch. On commit, the job runs, bringing down the latest code to the human test instance and running any outstanding updates. The job will fail if drush reports any errors in the update process, which is important as sometimes update functions can get muddy.

Using human test instances, a team can demo their new features or fixes to anyone in the organisation without having to get them into trunk first. This greatly improves the stability of trunk as incomplete or experimental features stay out of trunk in sandbox branches where they belong.

The instances are also used for training purposes, for editors to experiment with, and for the ad operations team to trial new ad formats on. Cheap and painless to spin up, very handy for everyone.