How we are using Jenkins and DigitalOcean

My main contribution to the Squid Web Cache project is these days running the project’s infrastructure. A lot of it is the project’s CI/CD farm.

In order to run it, we rely on a very kind donation by DigitalOcean . We use a VM hosted there to run the main jenkins instance and part of the jobs for the x86-64 architecture. We are then using the jenkins digitalocean plugin to spin up instances (droplets) on demand when we need to have more throughput from our build jobs.

In order to maximise how we use our resources, we rely on docker to run all of our target linux userlands. This allows us to decouple the runtime environment from the machine that’s running it, and to ensure consistency across builds (also coming up: a proper staging system).

In this post I’ll focus on how we spin up these instances, the whole setup is a bit more convoluted.

The digitalocean plugin is quite well integrated and easy to use; TBH I haven’t tried plugins for EC2 or GCP, but my other reference point, jclouds, was much harder to configure and set up.

Given our prerequisites, we need ondemand instances to only contain the docker runtime and java, which is needed to run the jenkins slaves as unlike other setups I’ve found online, these run outside the docker containers.

In order to do that, we supply to the “User Data” section this config snippet:

#cloud-config
apt_upgrade: true
package_upgrade: true
packages:
 - openjdk-11-jre-headless
 - docker.io
users:
 - name: name of the jenkins user on the executor machine
   groups: docker
   shell: /bin/bash
   ssh-authorized-keys: ssh-rsa ssh public key of the user jenkins runs under

These actions will be run when the droplet is launched, and prep the executor for jenkins to ssh into it and run the test jobs. In order to give the droplet time to do that, we need to wait for it with this init script:

#!/bin/bash

echo "starting init script"
while ! cloud-init status|grep -qF 'done'
do
  echo "waiting for cloud-init to be done"
  sleep 10
done

The next tricky bit is in the Droplet section, in the node Labels section we define a label for triggering the instance startup when needed. It can be anything, in our case docker-build-host, and an instance cap.

Referencing this label in the projects’ configuration matrix will trigger the spinup and imaging. Jenkins will then connect to the droplet via ssh and use docker run commands to test the various runtime environments

Advertisement

Converting from MoinMoin to MediaWiki

The Squid Wiki is hosted on an own instance of MoinMoin. We picked it at the time as it had fewer external dependencies than other engines, and it fit the bill.

Over time, and as the number of pages grew, its strengths became limitations, and I’m currently exploring whether to switch to a different engine. Mediawiki is the go-to choice for most people, so that’s what I investigated first.

W3C has developed a tool to convert from one to the other, but it hasn’t been updated in some time, to the point where MediaWiki API changes have bit-rotten it. It doesn’t help that this tends to be a one-off activity, so it

Open Source to the rescue! I have patched it to support current API and it worked for me ™. While waiting for the PR to be approved, feel free to use my fork of it

Squid’s in the Clouds

After the incident to the Project’s main server the Sysadmin team has started looking for ways to improve the reliability and performance of all of our services.

Our sponsor Rackspace has donated us a number of virtual machines, and we decided to leverage that for most of our core infrastructure. The main website, official source code repository, bugzilla and other essential services have since been migrated to virtual machines. Some of the benefits we expect to obtain are improved reliability, easier upgrades, simpler backups; we will not be abandoning physical nodes, there are some tasks that can’t be easily virtualized.

Our continuous integration testing farm used to run on a number of smaller VMs, but this was quite inefficient: smaller VMs would spend most of their time idle, and wouldn’t have much horsepower when active, resulting in relatively long build times. We currently test each commit on over a dozen different OSes and OS versions, across three different compilers. Each run requires over four and a half hours.

Docker has gained lots of mindshare in recent months as a lightweight, flexible and functional containerization platform, and after giving it a try I proposed to the project sysadmins to adopt it as the Project’s platform of choice for most of our build farm, on top of a beefier VM.
After a bit of testing, the results are very satisfying. Our main Linux farm infrastructure has changed from 12 single- or dual-core VMs to 1 8-core system running a dozen build-node containers. Build times are down by 25% and we are able to more easily overcommit CPU and memory resources; disk usage is dramatically down: while a full VM typically reqiures 40 GB of HDD space, a container only needs 4, half of which is ccache data.

As Docker containers are not visible to the network unless explicitly configured, security is up and we need less work to maintain them. It would be rather trivial to fully automate the deployment of newer versions of several Linux distributions by writing a few Dockerfiles.

Technically, most of this could be obtained by chroot(2) call and the right setup; we are only scratching the tip of the cloudy containerization iceberg. But being able to migrate a nontrivial setup in only a few hours of work could say that in the right hands the tools are already in a pretty advanced state.

Welcoming a Squid Project sponsor: Rackspace

Testing is a very important part of the life of a project such as Squid, especially as we wish to support as many different operating systems and platforms as we can, as efficiently as we can.
This also means that we can’t restrict to well-known POSIX APIs, but we need to account for each OS’s particular quirks and optimizations, especially in those areas where there is the need of a deep integration with the OS.

One of the tools which we have added a few years ago is the build farm. When new code is committed to a followed branch, our Jenkins setup will notice it, and schedule a test run on all the eligible nodes. This means we need to rely on a mix of operating systems, libraries and compilers.

We are always on the lookout for sponsors, both individuals and companies, and back in July Rackspace volunteered to donate space for the Squid project us on their public cloud infrastructure. As of now, we have 11 virtual machines and one database hosted in their cloud, running mostly various flavors of Linux and FreeBSD. The experience has been very satisfying, in particular I’ve found the claim about “fanatical support” to be well-founded in the dedication and kindness of everyone who was involved in making this happen.

It was not all rose petals, of course. Our needs mean that we are better off with something resembling the bare metal, while the model Rackspace offers is more oriented towards having predefined images which configure certain aspects such as network addresses, admin users’ passwords etc.

Still, it is very refreshing that if an OS is among the list of supported images (as an example, Ubuntu Saucy was available about three days after Canonical released it) setting up a new build node in the farm takes literally 15 minutes, most of which is spent waiting for the server image to be cloned from the template.

The first effects of the newly expanded build farm are already apparent: we have reached a stable build status on a wide number of Linux, FreeBSD and OpenBSD versions, with gcc, clang and icc compilers. We are starting to set sight on MacOSX and to bring Squid 3 to Windows. All of this would have been much harder, or even impossible, without the Build Farm.

So on behalf of everyone who works on Squid, I’d like to thank Rackspace for their kind contribution to the Project.