How we are using Jenkins and DigitalOcean

My main contribution to the Squid Web Cache project is these days running the project’s infrastructure. A lot of it is the project’s CI/CD farm.

In order to run it, we rely on a very kind donation by DigitalOcean . We use a VM hosted there to run the main jenkins instance and part of the jobs for the x86-64 architecture. We are then using the jenkins digitalocean plugin to spin up instances (droplets) on demand when we need to have more throughput from our build jobs.

In order to maximise how we use our resources, we rely on docker to run all of our target linux userlands. This allows us to decouple the runtime environment from the machine that’s running it, and to ensure consistency across builds (also coming up: a proper staging system).

In this post I’ll focus on how we spin up these instances, the whole setup is a bit more convoluted.

The digitalocean plugin is quite well integrated and easy to use; TBH I haven’t tried plugins for EC2 or GCP, but my other reference point, jclouds, was much harder to configure and set up.

Given our prerequisites, we need ondemand instances to only contain the docker runtime and java, which is needed to run the jenkins slaves as unlike other setups I’ve found online, these run outside the docker containers.

In order to do that, we supply to the “User Data” section this config snippet:

#cloud-config
apt_upgrade: true
package_upgrade: true
packages:
 - openjdk-11-jre-headless
 - docker.io
users:
 - name: name of the jenkins user on the executor machine
   groups: docker
   shell: /bin/bash
   ssh-authorized-keys: ssh-rsa ssh public key of the user jenkins runs under

These actions will be run when the droplet is launched, and prep the executor for jenkins to ssh into it and run the test jobs. In order to give the droplet time to do that, we need to wait for it with this init script:

#!/bin/bash

echo "starting init script"
while ! cloud-init status|grep -qF 'done'
do
  echo "waiting for cloud-init to be done"
  sleep 10
done

The next tricky bit is in the Droplet section, in the node Labels section we define a label for triggering the instance startup when needed. It can be anything, in our case docker-build-host, and an instance cap.

Referencing this label in the projects’ configuration matrix will trigger the spinup and imaging. Jenkins will then connect to the droplet via ssh and use docker run commands to test the various runtime environments

Advertisement

Converting from MoinMoin to MediaWiki

The Squid Wiki is hosted on an own instance of MoinMoin. We picked it at the time as it had fewer external dependencies than other engines, and it fit the bill.

Over time, and as the number of pages grew, its strengths became limitations, and I’m currently exploring whether to switch to a different engine. Mediawiki is the go-to choice for most people, so that’s what I investigated first.

W3C has developed a tool to convert from one to the other, but it hasn’t been updated in some time, to the point where MediaWiki API changes have bit-rotten it. It doesn’t help that this tends to be a one-off activity, so it

Open Source to the rescue! I have patched it to support current API and it worked for me ™. While waiting for the PR to be approved, feel free to use my fork of it