Continuous Integration

For the last few years there has been a slow growing improvement to the testing and QA Squid is subject to. This last week has seen the construction and rollout  of a full-scale build farm to replace some of our simple internal testing. Robert Collins covers the growth process in his blog.

Here is the initial release notice:

Hi, a few of us dev’s have been working on getting a build-test environment up and running. We’re still doing fine tuning on it but the basic facility is working.

We’d love it if users of squid, both individuals and corporates, would consider contributing a test machine to the buildfarm.

The build farm is at http://build.squid-cache.org/ with docs about it at http://wiki.squid-cache.org/BuildFarm.

What we’d like is to have enough machines that are available to run test builds, that we can avoid having last-minute scrambles to fix things at releases.

If you have some spare bandwidth and CPU cycles you can easily volunteer.

We don’t need test slaves to be on all the time – if they aren’t on they won’t run tests, but they will when the come on. We’d prefer machines that are always on over some-times on.

We only do test builds on volunteer machines after a ‘master’ job has passed on the main server. This avoids using resources up when something is clearly busted in the main source code.

Each version of squid we test takes about 150MB on disk when idle, and when a test is going on up to twice that (because of the build test scripts).

We currently test:

• 2.HEAD
• 3.0
• 3.1
• 3.HEAD

I suspect we’ll add 2.7 to that list. So I guess we’ll use abut 750MB of disk if a given slave is testing all those versions.

Hudson, our build test software, can balance out the machines though – if we have two identical platforms they will each get some of the builds to test.

So, if your favorite operating system is not currently represented in the build farm, please let us know – drop a mail here or to noc @ squid-cache.org – we’ll be delighted to hear from you, and it will help ensure that squid is building well on your OS!

-Rob

That just about covers everything. Hardware and build software requirements are listed in the build farm page.

Hi, a few of us dev's have been working on getting a build-test
environment up and running. We're still doing fine tuning on it but the
basic facility is working.

We'd love it if users of squid, both individuals and corporates, would
consider contributing a test machine to the buildfarm.

The build farm is at http://build.squid-cache.org/ with docs about it at
http://wiki.squid-cache.org/BuildFarm.

What we'd like is to have enough machines that are available to run test
builds, that we can avoid having last-minute scrambles to fix things at
releases.

If you have some spare bandwidth and CPU cycles you can easily
volunteer. 

We don't need test slaves to be on all the time - if they aren't on they
won't run tests, but they will when the come on. We'd prefer machines
that are always on over some-times on.

We only do test builds on volunteer machines after a 'master' job has
passed on the main server. This avoids using resources up when something
is clearly busted in the main source code.

Each version of squid we test takes about 150MB on disk when idle, and
when a test is going on up to twice that (because of the build test
scripts).

We currently test
2.HEAD
3.0
3.1
3.HEAD

and I suspect we'll add 2.7 to that list. So I guess we'll use abut
750MB of disk if a given slave is testing all those versions.

Hudson, our build test software, can balance out the machines though -
if we have two identical platforms they will each get some of the builds
to test.

So, if your favorite operating system is not currently represented in
the build farm, please let us know - drop a mail here or to noc @
squid-cache.org - we'll be delighted to hear from you, and it will help
ensure that squid is building well on your OS!

-Rob

Life of a Beta

From early inception when the developers have nothing but dreams for it.  Through the coding and arguments about what should be included and how. Through the alpha testing with its harrowing hours pondering obscure code from last decade. Even the odd period of panic as security bugs are whispered about behind closed doors. Such is the early life of software.

Two weeks ago word went out that 3.1 was reaching end-game.

This part of the release lifecycle seems to be going well. Packages appearing very slowly as QA throws demanding eyes on the code and making us actually fix things. Don’t be fooled by the packages out already, they have been in QA for a few months to get this far. On that note:

NetBSD, Gentoo, Ubuntu, FreeBSD and RedHat already have packages ready and available for at least testing use if you know where to look (ie the links right there might be a good start).

Debian has a bit more QA to go as of the writing, but the maintainer tells me there will be packages out soon.

OpenBSD and Mac turned out at the last minute to be running split-stack IPv6 implementations (for security apparently). All the documentation read in two years left the impression it was a Windows XP anarchism (and who runs XP Pro on a server?), so support was delayed and delayed.  The OpenBSD maintainer and someone interested from Mac are working with myself on closing that gap in the features.

There may be more OS with 3.1 packages. I’ve only begun working my way down the distrowatch.org popularity list to see which OS do and who to contact. Squid has bundles on over 600 OS apparently.

If you know who does the official packaging for your OS and whether there are 3.1 packages ready, please do me a favor and mention it. I’m seeking a web page where to find the squid (or squid3/squid30/squid31) package information and also the place where distro bug reports about Squid might end up.

Release 3.1

Kinkie pointed out Linus Torvalds blog today to the rest of us here working on Squid. As the release maintainer for Squid-3 this year I kind of agree, its a sad time to cutting a new version. For me its more of a reflection that for all the high hopes we have of this new release, we had the same or similar hopes of the earlier one. Just 12 months ago now.

On that sad note, yes its finally happened. 3.0 has aged into a full blown stable package. Most of a month and no new bugs. Perfect time for something shiny and new for the neo-tech fanclub. And so with that for an intro we are gone for 3.1 !

3.1 is available for beta testing in the form of 3.1.0.1. see the Release Notes for further details on the finer details of change.

This release has gained from the experiences of 3.0 and 2.6, starting from a much more stable base of code than the initial. 3.0 had a long period of years with few active developers, an interminably long period of testing releases, and in hindsight a premature birth.

Alongside the code this release has a wider collaboration with active users. For the first time in many years we held a Developer meeting that included Users. We who were there certainly took in a lot of feedback from all sides. I hope those users who talked to us can see in this release that their comments, even those made in passing, have been listened to and worked on.

The small comment from one user when asked what their biggest itch with squid was “we don’t like these being called STABLE, when its obvious they are not.” has led to the most notable change made to 3.1.  That comment and similar feelings by others lead us into discussions on the release naming and numbering. From which we have produced – 3.1.0.1 – the second milestone point of the branch we are calling 3.1. Where the developers have everything done and working for us.

no more DEVEL, PRE, or RC, no more premature labels guessing when things might be STABLE.  Just 3.1.0.1. Further testing from the rest of you will show whether anyone can consider it stable, unstable, usable or as buggy as raw earth.

From the developers; We use it. We love it. Try it, and see for yourselves.

Some of the stuff you will find there is;

• a lot of small changes aimed towards easier use and configuration (three cheers to those who nagged long an hard for this).
• a lot of network RFC compliance extensions, making 3.1 much more capable of meeting modern network needs. The future still holds improvements, but 3.1 is definitely better in many respects than everything that came before.
• a lot of things to make Squid a better experience for your own users. More seamless network recovery tricks than ever before. We have even tagged along behind the international localization bandwagon in our own way to make the errors squid does have to show both pretty and readable.

Sadly, careful readers will notice a section of the Release Notes labeled “Regressions against 2.7″.  Yes, those of you who moved to 2.7 because you needed some brand new feature there may still have trouble migrating up to 3.1. What we have done is to port as many of the 2.6 features and fixes as we could. A few did not make it in time, but will be coming in 3.2, alongside the features added as experimental in 2.7.

On the overview:

• 2.5 has disappeared over the horizon into the long dark night of obsoletion.
• 2.6 is itself officially aging out now. Supported, but the developer first response is “can you try something newer?”.
• 2.7 is being maintained for the few extremely high-performance accelerator setups. But in general the Squid-2 sequence is aging out for us developers.
• 3.0 has reached a point of stability, though not fully-featured.
• 3.1 is available for testing as the next step up. You should be planning to migrate up to 3.1 or later release.

If there are any features holding you to Squid-2, or even an issues you find with testing Squid-3 speak up, we rely on your input to choose the most needed features for porting.

Thank you all, and enjoy your use of Squid 3.1

Squid-2.6 + TPROXY + Debian

Jason Healy posted some useful information to the squid-users list a week or so ago.

Quoting:

I’ve been a happy user of Squid for the past 10 years or so, and I’d like to take a second to thank everyone who has worked so hard to make such a great piece of software!  I’d like to give back to the Squid community, but unfortunately I’m not much of a C hacker.  However, I’m hoping I can still help.

I’ve just spent a few days getting my school’s Squid install up to date (we were running 2.5 on Debian Woody).  I switched to using tproxy this time around (we used to do policy routing on our core, but it was spiking the CPU too much).  Thanks to the mailing list, some articles on the web, and a little messing around I was able to get the whole system up and running.  I’ve documented the steps here:

http://web.suffieldacademy.org/ils/netadmin/docs/software/squid/

The document is written for someone with a decent grasp of Linux, and is specifically geared to Debian Etch.  There are some tweaks that are pecific to our install (compile-time flags, mostly), but otherwise it’s pretty generic.  Hopefully, this will help someone else out who’s trying to build a similar system, so I’m posting so it will hit the archives.

Squid Updates – April 2008

University studies have begun for me and so my available time has been limited. But to summarise:

• Squid-3.0 has been released, for people who are interested in playing with it
• Kinkie has updated the Wiki theme in a big way – http://wiki.squid-cache.org/
• Squid-3 development has migrated to bzr
• Alex is looking to merge in the first set of eCAP related changes into Squid-3.HEAD
• Squid-2.7 is on track to be released – there’s one outstanding bug and its unfortunately difficult to fix. http://www.squid-cache.org/bugs/show_bug.cgi?id=2160 is the bug to watch.
• Funded Squid-2 development will continue for the time being; mostly from projects I’m working on. We’ll see how things progress there. The Squid-2 Roadmap is slowly changing, evolving and being completed.

Squid-2 performance work: graph #1

 

Whats going on with Squid-2 and Squid-3 ?

A few people have asked me what the deal is with Squid-2 and Squid-3.

“Why are you developing on Squid-2 when Squid-3 is now out?”

“Should I upgrade to Squid-3 now that its released?”

I’m focusing on Squid-2 for a few reasons, namely:

• Its what people running high-traffic sites are currently running, and Squid-3 doesn’t work at all for them;
• I was fed up waiting for Squid-3 to be released and for it to become mature enough for users to migrate to before I started my performance work. I gave up about 12 months ago and began planning out the work thats currently going on.
• I’m personally much more familiar with the Squid-2 codebase than the Squid-3 codebase.

So what exactly am I doing to Squid-2? Well, I’m doing all the things to Squid-2 which I personally believe we should’ve done in the C++ Squid-3 branch before all the “new stuff” was added. You can find it all at http://devel.squid-cache.org/changesets/squid/s27_adri.html . A summary of what I’m doing in this first round:

• I’m taking a very sharp scalpel to the codebase and removing all of the extra data copies and buffering which is going on;
• I’m reworking the buffer management so arbitrary sized data buffers can be used, rather than fixed 4k buffers for network/disk traffic;
• I’m reworking the Strings interface to use reference counting and reference underlying buffers, saving on memcpy() and malloc() calls, cutting down on the amount of transient memory used to handle requests and dropping the CPU and memory bus utilisation quite dramatically;
• I’m reworking the dataflow between server->store and store->client to use the above reference counted buffers, so data isn’t memcpy()’ed between layers, again dropping CPU and memory bus utilisation;
• And I’m going to break out as much of the code into external libraries with well-understood dependencies, as preparation for documentation, unit testing and further profiling.

My aim is to fix whatever bugs show up in Squid-2.7 and then in Squid-2.HEAD (which has some of the above included already.) I’ll then start bringing across my changes as they’ve been tested and been found stable. My aim is to have the bulk of the above done within the next month or so and get it into Squid-2.HEAD and concentrate on making it stable before I continue tidying up the dataflow and restructuring the ugly bits of code.

Whats this mean for Squid-3? The Squid-3 guys are doing some great work with things such as ICAP and IPv6 and I hope that they’ll gain more experience with their codebase over the next 12 months or so. I’m certainly not bringing ICAP support into Squid-2 until I’ve reworked the dataflow and tidied up the code enough for ICAP to sit comfortably in the data pipeline, rather than have it bolted onto the side and hooking into strange places where it shouldn’t. (I may bring in IPv6 into Squid-2 soon though!)

Hopefully my work and their work will culminate with the development of the next Squid major version over the next 12 to 24 months. There’s a long way to go though and my main aim here is to get faster, better and shinier code out to the majority of Squid users now so they can benefit from the development, rather than repeating the 4-odd year gap between Squid-2.5 and Squid-2.6. Users hated that.

So whats it mean for you?

• If you want to try out Squid-3; if you want supported ICAP services then try it out.
• Squid-2.X will continue being developed over the next 12 months as time permits, so don’t feel like you have to move to Squid-3.
• If you feel adventurous, try out Squid-2.7. Initial reports are that its stable and slightly less CPU intensive.
• Squid-2.7 is the first version to include changes to allow Youtube and Microsoft Updates caching. It doesn’t do it out of the box, but the support is there, and I’ll be publishing test rules soon to let people start caching this stuff.
• If you feel really adventurous then try out Squid-2.HEAD and report back if you have any issues. It should be even less CPU intensive, but only under certain workloads.

Please upgrade to Squid-2.6.STABLE18

Squid-2.6.STABLE18 fixes a silly bug (thanks to yours truely fixing another bug!) which may cause your Squid to crash under certain circumstances.

Squid-2.6.STABLE18-RC1 (release candidate 1) tarballs are available from the Squid website – http://www.squid-cache.org/Versions/v2/2.6/ – the release should be in a day or two.

Squid-2.7 Branched; performance work has begun!

Henrik has branched Squid-2.7 – it hasn’t been formally announced yet but it should be any day now.I’ve begun rolling in infrastructure changes with an eye towards improved performance in Squid. Squid-2 is my testbed at the moment – I’m leaving Squid-3 alone for now to let the codebase mature and the C++ guys to, well, do their C++ “thing”. The first round of patches to Squid-2.HEAD remove one of the major CPU and memory bottlenecks – memcpy()’ing of data as it passes from the store (so from anywhere, really) back to the client. This may or may not improve performance with your workload but its the beginning of sensible dataflow inside Squid.(I estimate this brings Squid up to the late 90’s in terms of network application coding..)My next trick will be reference counted buffers and strings, to avoid more memcpy()ies, memory allocation/frees, and general L2 cache busting. More on that later. 

Squid-3.0.STABLE1 released

Its been a long wait, but Duane has released Squid-3.0.STABLE1. Features include integrated ICAP support. You can find more information at the release website.