Whats going on with Squid 4

Those that have been paying attention will have noticed that Squid-4 beta cycle has been going on for a very, very long time. A whole year now in fact. There are several reasons for this.

* The ecosystem that Squid is deployed into has a somewhat mixed situation in regards to C++11.

Even 5 years after it was standardized compiler support is still not readily available in some popular OS distributions. In particular there are a large number of people clinging to the outdated but still supported RHEL 6 and its derived family of OS which do not easily provide recent versions of GCC. So we are procrastinating on the deprecation of Squid-3.5 to give more people a chance to move on. The clock is ticking though.

* We had a lack of early adopters testing the early versions of Squid-4.

So bugs that only show up in real usage have been very slow to be found. We don’t make a new version start its beta cycle until the developers are reasonably sure that its stable enough to be used. The beta process is supposed to be just a confirmaton of that. Indeed a year ago 4.x looked like it had no bugs at all. Which is kind of suspicious, but not impossible. As the betas progressed though the bug reports started rolling in. It is somewhat a testament to how our modern build systems are catching out minor and trivial bugs that these tester reported issues have largely all been difficult to resolve.

* We are experimenting with a new management process for Squid releases.

Previously Squid-5 would have been branched for development and Squid-4 starting to stagnate, er, become “stable”. This time though we have delayed the branching and instead kept Squid-4 as main development branch for minor changes while the beta cycle goes on. That has made it a little more volatile than most of the Squid-3 series betas.

All up, we are down to the last few major bugs to be resolved in the new code. Progress on those is slow but steady. So Squid-4 production “stable” release should be not to far ahead on the calendar.

Advertisement

Squid-3.2 mythbusting NAT

One of the more frequently mentioned “problems” with Squid-3.2 since its release is a change in how it handles NAT failures.

The Myths

“Squid used to work when I NAT traffic to it from my router.”

“Squid used to work with one port when I configure the browser and NAT traffic.”

The Reality

No. Squid up until 3.1 would silence the NAT errors and treat the router as if it were the client browser. Any differences between the Host header and the requested URL were also completely ignored. All of this would be invisible to you the administrator, hidden at debug levels not normally shown. With several of the problems being completely unrecorded as well.

This last part seems to be behind a fair bit of skepticism about whether the problem we solved really exists. Nothing was showing up in the logs before and even a full packet trace from Squid or a gateway server would not reveal how JavaScript hijacks a client browser to pass internal documents to an external attacker.

What Changed?

Squid-3.2 finally added Host header validation to protect against the nasty behaviour and a few 0-day attacks using CVE-2009-0801 security vulnerability. Which has been a known flaw with NAT interception by HTTP proxies for at least 12 years now. This meant two things had to change:

* ignoring NAT errors had to stop. We can’t rely on validation results if the TCP details are already known to be corrupted.

* traffic directly from a browser to a NAT intercept port on Squid had to stop. There is no way to separate the NAT lookup result of error and unknown entry. A pity really, but that is what we are forced to work with.

During testing we uncovered a lot of really quite nasty behaviour by various clients, and indeed some public services setup to take advantage of the NAT bug as if it were a feature. On the whole a lot of client software sends a Host header with values strangely unrelated to what is being requested of the proxy. All of this had to go, or did it? for most of the year or so of testing it was banned completely and a HTTP error message returned for any such garbage. But in the end it was clear that we had to let it through somehow or we would be pitting Squid against the biggest players on the Internet … yeah.

(Almost-) Final Results

Squid-3.2 and later will reject traffic where NAT lookups fail on an intercept port. This includes NAT done on external devices and browsers directly sending proxy requests to the Squid intercept port. At the very least accurate reporting about the traffic and what it contains is the critical factor. If we let this traffic into Squid it would seriously compromise the validation results and also allow malicious clients to attack internal network resources with a large measure of anonymity. Neither of which is acceptible for a proxy trusted with controlling user traffic.

The best practice guideline for some years has been that NAT MUST be done on the Squid device. Squid-3.2 are now enforcing it as a basic requirement. If you are one of the network administrators running into this requirement change, please investigate the Policy Routing functionality of the device you originally had doing the NAT.

Squid-3.2 and later will accept invalid Host headers and produce a response to the client. But they will not cache the untrustworthy transaction results. They will contact the same server which the client TCP connection would have originally reached had Squid not been there (visible as ORIGINAL_DST in the logs). Notice that this is now deserving of the name “transparent proxy” a lot more than previous intercepting Squid. Even so NAT interception is still only half-transparent with the server being able to easily identify the proxies existence.

The loss of caching is intended to be a temporary solution until we can properly implement per-client caching of objects. As the saying goes there is nothing so permanent as the temporary – Squid still contains this workaround several series later. Support with this work is very welcome.

The situation when upstream peers are involved is also quite dangerous. For now we have had to permit Squid to pass the traffic to peers, which opens all multi-hop systems of proxies to the same vulnerabilities that were previously possible wth a single intercepting proxy. The solution is also going to require a substantial amount more work and be some years away. For the meanwhile it is a good idea to avoid passing intercepted traffic to cache_peer when possible.

More information on the problems, log entries, when they occur and what can be done about each is detailed in the Squid wiki Host header forgery page.

Zero-Sized Replies from Windows Servers

During the last few months there have again been a number of bug reports and queries from administrators seeing Zero Sized Reply error pages being produced by Squid 3.2 and later.

These “errors” are produced when Squid sends an HTTP request, then something out in the network goes wrong and the TCP connection gets severed while Squid is still waiting for the start of HTTP response to arrive. As you can imagine this is a little vague because that “something” is any one of a large set of potential networking problems.

Investigation of the old usual culprits in ECN, Window Scaling, PMTUd, and CONNECT proxying ruled them out leaving us mostly in the dark.

Testing without the proxy appeared to work fine. As did small short transactions even through the proxy. Leaving us more than a little confused.

The most common theme this time seems to be Windows based SSL/TLS services with recent but not top of the line software versions. IIS or Sharepoint on Server 2008 and 2010 for example.

Daniel Beschorner has done some investigating and reported this:

Since Squid 3.2 the SSL flag SSL_OP_ALL is no longer enabled by default in Squid. It enables different workarounds in the OpenSSL library.

Windows / IIS seems to get confused by empty packets (to mitigate the BEAST attack) sent from OpenSSL in TLS 1.0.

So the possibilities are:

• http://support.microsoft.com/kb/2634328 (patch your Windows)
• use TLS>1.0 (BEAST-safe by design)
• disable BEAST mitigation by ssloptions=ALL in squid.conf (insecure)

We have also had remarkably similar problem reports about iTunes servers. That one is still unconfirmed and unresolved.

Squid-3.2: managing dynamic helpers

One of the new features brought in with Squid-3.2 is dynamic helpers. A brief name for a very useful administrative tool and like all tools can be both easy and tricky to use at the same time.

If you have a proxy using helper processes but only a small cache (or none) this is a feature for you.

The good news

Configuration is super easy – just set the initial startup, maximum number of helpers and an idle value for how many to start when new ones are needed to handle the request load.

Dying helpers have a higher threshold before they kill Squid. It is not perfectly tuned yet in the code so improvements will contnue to happen here, but already we see small bursts of helper failures being suppressed by re-started replacements without that all too familiar Squid halt with “helper dying too quickly” messages. Note that this is just higher, not gone.

The bad news

Determining what those values should be is no more easy or straightforward than before. Squid uses fork() to start new helpers. The main side effect of this is that helper instances started while Squid is running will require a virtual memory size equivalent to the Squid worker process memory at the time they are started. If your proxy is pushing the box to its limit on RAM, dynamically started helpers could easily push it over to swapping memory at the worst possible time (peak load starting to arrive). Also on the bad news side is that the helpers are run per-worker. Which has the potential to compound the RAM usage problems.

We do have a proposal put to the development team which will almost completely remove this problem. Having the coordinator or a special spawner kid do the forking instead of the heavy workers. But as of this writing nobody is working on it (volunteers welcome, please contact the squid-dev mailing list).

Practice Guidelines

While it may look like the bad news is worse than the good news it does turn out that most installations are small instances or non-caching worker proxies these days. All of which may need lots of helpers, but are not heavy on the RAM requirements. For all these installations dynamic helpers are ideal and in a lot of cases can even be set with zero helpers on startup for a very speedy delay to first request accepted time.

The caching proxy installations with higher memory requirements in the workers can still make use of the dynamic nature to avoid complete outages in the worst-case situations where peak traffic overloads the planned helpers. But should normally be configured as before with enough helpers to meet most needs started before the RAM requirements become too onerous on the worker.

Until at least the bad news problems above are resolved the default behaviour for Squid will continue to be starting all the maximum helpers on startup. So there are no unexpected surprises for upgrading, and the old advice on calculating helper requirements is still useful for determining that maximum.

Squid-3.2: Pragma, Cache-Control, no-cache versus storage

The no-cache setting in HTTP has always been a misunderstood beastie. The instinctual reaction for developers everywhere is to believe that it prevents caching or cache handling or some such myth.

This is not true.

By definition it merely forces caches to revalidate existing content before use (ie it tells the proxy to “be ultra, super-duper conservative. Do not send anything from cache without first contacting the server to double check it.”).

When sent on a client (browser) request:

• Pragma:no-cache instructs HTTP/1.0 caches to revalidate any cached response before using it.
• Cache-Control:no-cache instructs HTTP/1.1 caches to revalidate any cached response before using it.
• Pragma:no-cache only works for HTTP/1.1 caches when Cache-Control is missing.
• all other values of Pragma are undefined and to be ignored.

When sent on a server response:

• Pragma in all its forms has no meaning whatsoever. It must be ignored.
• Cache-Control:no-cache instructs HTTP/1.1 caches to revalidate this response every time it is re-used.

If you read those bullet points above very carefully you will notice that at no point is store mentioned. None whatsoever. The closest it gets is mentioning what to do with already-stored content (revalidate it). In fact the HTTP/1.1 specification goes as far as to say explicitly that responses with no-cache MAY be stored – provided the revalidation is done as above.

no-cache in Squid

The well-known squid versions of the past have all been HTTP/1.0 compliant and advertised themselves as HTTP/1.0 software. These proxies both looked for Pragma:no-cache headers and obeyed them:

• Squid being HTTP/1.0 that Pragma took precedence over Cache-Control.
• Due to lack of full HTTP/1.1 revalidation in very old versions Squid has traditionally treated no-cache in either header as if it were Cache-Control:no-store.
• Due to some old server software Pragma:no-cache on responses was treated as a mistaken form of Cache-Control:no-store.

Starting with version 3.2 Squid is advertising and attempting to fully support HTTP/1.1 specifications. This is a game changer.

All of the above is about to be up-ended, assumptions can be thrown away and some funky cool proxy behaviour allowed to take place.

Hiding in the background is the instruction that Pragma only applies when Cache-Control is missing from a request. We can ignore it – almost completely. When we do have to pay attention we only need to notice the no-cache value and can treat it as if we received Cache-Control:no-cache.

The other change is a potential game changer: The object being transfered is stored now, revalidated later.

Some implications from storing no-cache responses:

• servers can utilize 304 responses instead of generating new content. Saving a lot of bandwidth and CPU cycles.
• all those configuration hacks for ignoring or stripping no-cache are no longer needed. Also, the harm they do will become more visible as revalidation is skipped.
• cache HIT ratio can potentially rise above 50% for forward proxies. As a side effect of the HIT counting market a large portion of web traffic is utilizing no-cache instead of no-store or private. This large portion is cacheable but until now Squid has been dropping it.

Before the marketing department panics about the end of the world lets be clear on one important point:

revalidation means every client request will still reach the end server doing HIT counting, traffic control, whatever – but in a way which allows 304 bandwidth optimization on the responses.

Do not expect a sudden rise of TCP_HIT in the proxy logs though. It is more likely to show up as TCP_REFRESH_HIT or the nasty TCP_REFRESH_MODIFIED/TCP_REFRESH_MISS which is produced by broken web applications always sending out new unchanged content.

Proxying HTTPS for an internal service

Since version 2.6 changed the way http_port worked and let Squid service multiple different types of traffic simultaneously people have been struggling with one setup which should to all outward appearances be quite simple.

I’m speaking of the scenario where you have a proxy serving as both a forward-proxy gateway for the internal LAN users and as a reverse-proxy gateway for some SSL secured internal services (an HTTPS internal site).

Both setups are essentially simple. For the reverse-proxy you setup an origin cache_peer with SSL certificate options. Perhapse an https_port to receive external traffic.  For the forward-proxy you setup users browsers to contact the proxy for their HTTP and HTTPS requests. Perhapse with NAT interception to force those who refuse.

They you discover that Squid can’t seem to relay requests from internal users to your internal peer. You get warnings about clientNegotiateSSL failing on plain HTTP requests. Even though it may appear the user was opening HTTPS properly to contact it.

The problem is that when relaying through a known proxy browser wrap their SSL request, inside a CONNECT tunnel setup request. Which is plain-text HTTP. Squid passes this intact on to any cache_peers you have configured. Even the origin one which is expecting SSL. It may do the right thing and wrap it in a second layer of SSL. But that just makes things worse as the server at the other end gets this weird CONNECT request it cant do anything with.

Until recently the only fix has been to setup a bypass so that internal LAN users don’t use the proxy when visiting the internal HTTPS site. Which works perfectly for user access. But does cause problems on the recording and accounting systems which now have to track two sets of logs and filter proxy relayed requests out of one.

Or alternatively to set the LAN DNS to point users at the reverse-proxy port and figure some way to avoid forwarding loops by bypassing Squid like above or disabling the loop detection.

Both alternatives having the same problems at best. Worst case in the second you have opened some security vulnerabilities by ignoring loops.

In Squid-3.1 we have trialled two possible ways to fix this whole situation.

The first attempt was to simply not relay CONNECT to peers with origin type configured. This failed with a few unwanted side effects. One was that Squid would lookup the DNS and go to that server. Fine for most, but not all Squid have split-DNS available. Or Squid could relay it to a non-origin peer instead. Possibly halfway round the world with worse lag effects than a little extra calculation handling the logs.

The second attempt, which we are currently running with in 3.1.12 and later. Is to strip the CONNECT header and connect the tunnel straight to the peer. But only when the peer port matches the intended destination of that tunnel, and your access controls permit it for selection.

• The port restriction is there as a simple check that the service is likely to match protocols. Even if we cant be sure which.
• Traffic to that internal service does go through the proxy and traffic accounting only has to handle the proxy logs.
•  Requests from LAN clients use the clients SSL certificates instead of the cache_peer configured ones.

This last point is one which can bite or confuse. If you have LAN users in this type of scenario and require all contact with the internal service to use the proxy configured certificates you will still need to configure those clients with the old methods.

 

Enjoy. And as always, if you have better ideas or problems please let us know.

Squid Proxy Server 3.1: Beginner’s Guide

For those who have been waiting and asking there is now a beginners guide to Squid-3.1 available for sale from Packt Publishing. Authored by Kulbir Saini.

This book seeks to be an introductory guide to Squid and specifically to the features available in the Squid-3 series so far. It covers both the basics and tricky details admin need to be aware of and understand when working with Squid. From configuring access controls through deployment scenarios to managing and monitoring the proxy operation this book has it.

It does not seek to be an update to the O’Rielly book, so there are many fine details and advanced technical descriptions missing. Although even experienced Squid admin may find new topics and features mentioned here that they were unaware of.

Disabling Authentication

As you may know Squid proxy builds and runs in some pretty interesting places. Amongst these is the WRT operating system. Although admittedly getting Squid small and light enough to run there is a major miniaturization job all by itself and its size still prevents other deserving applications from being used.

One of the problems that keeps popping up in this miniaturization effort is the  –disable-auth option. One would naturally expect such an option to disable authentication in Squid. It does not. So far all it does is prevent the additional helpers being built, and that only in the 3.2 series.

This month my spare-time task has been to make that directive work. To actually omit from the Squid binary as much code as possible which processes and manipulates the authentication information.

So far progress is looking good, around 900KB of the default binary size is removed.  100KB or so of run-time footprint. There are some not so obvious side effects of removing authentication which I’m going to cover here.

These changes have already landed in 3.HEAD and will be present in the 3.2 betas shortly.

In the security industry there are three terms: identification (of who or what something is), authentication (that some identification is true and correct) and authorization (that someone or thing is allowed to perform an action). Almost all access through Squid has a number of features acting to test these in various combinations. Luckily most of it takes the form of simple identification and authorization.

Authentication and ACLs

Naturally this being the goal of the project.

–disable-auth removes auth_param directives and also all ACL types which process usernames. Other than the ident username ACL which does not involve actual authentication.

IDENT

This protocol is all about identification and authorization. No authentication involved. As such its use has not been impacted in any visible way in the current Squid. However I’m noting it here since another side project underway is unifying the username details together.

–disable-auth removes the shared username objects and may in future affect storage of IDENT results.

External ACLs

External ACL helpers are permitted to perform what is called side-band authentication and return to Squid the username and password for an authenticated user. Since this is a form of authentication it dumps results into the username structures at various points.

–disable-auth will disable and remove the storage of these usernames as a possibility. Effectively disabling this type of authentication despite any support which may remain in the helper.

FTP

FTP protocol requires authentication credentials to be passed to the FTP server. Squid has several ways of doing this. Anonymous credentials are set in squid.conf and tried first. The URL may be sent containing an unsecured username and password in clear text.

Alternatively recent versions of Squid will look for HTTP Basic authentication credentials to use. Lacking any working credentials Squid will normally now reply with an HTTP authentication request, resulting in the user being able to enter their FTP login into a popup box or browser password manager.

–disable-auth or even just –disable-auth-basic will now prevent the HTTP authentication method from working in Squids FTP gateway.

Cache Manager

Squids HTTP management interface used by cachemgr.cgi, squidclient and other tools to fetch reports and perform administrative actions relies on authentication for certain actions.

Despite the fancy action@password style its URLs use this is converted into a real HTTP login header by the tools. The management interface relies on this authentication for all actions marked as protected.

–disable-auth or even just –disable-auth-basic will now prevent the Squid cache manager interface from receiving these credentials. Effectively blocking all access to the protected actions.

Delay Pools

The fancy new delay pools in Squid-3 rely on user credentials for some of their bucket types. If one thinks about it closely it becomes clear that without any authentication such pools are useless.

–disable-auth will prevent the class 4 delay pool from being built and available.

Peering

The cache_peer login directive which may normally pass on credentials to a peer via HTTP authentication headers has a complex and mixed relationship with authentication.

–disable-auth prevents login= from generating any HTTP headers. Since the auth encoding is disabled. However the PASSTHRU will still work. PASS is reduced to an exact equivalent of PASSTHRU since it may not generate a header for external ACL authenticated users.

–disable-auth-basic prevents login=username:password from working since the basic authentication header creation is disabled. However the NEGOTIATE and PASSTHRU remain working. PASS is reduced to an equivalent for PASSTHRU since the external ACL authenticated users require Basic encoding.

–disable-auth-negotiate prevents login=NEGOTIATE working since the negotiate authentication header creation is disabled. However all the other forms still work.

Project Update: Bugs, bugs, betas and breakages

So much for a monthly update with hints and tips. Been a year and more! So I guess this post should start again by covering what has been going on over this past year. “Not much and far too much” just about covers it …

 

Bugs …

Business as usual has seen a steady parade of upgrades from earlier versions up to 3.0 and 3.1. Bringing with it a steady stream of bug reports for all sorts of varied things overlooked or not yet fixed. The increasing number of reports people are giving of old bugs that have annoyed them without being reported is pleasing in a way. I hope this means less overall and display stuff is relatively easy

Some people have had shocks; bugs reported over a year ago being fixed suddenly and some new reports not lasting the day. I can’t stress enough how important your help in researching the problems is.

The oldest bugs are simple “Squid is doing X”. Um, yeah well, we can assume what the expected behaviour was (sometimes). After prodding (a day or so later) up comes a version number and build options.

For example; I was looking at a typical old bug today. Still open from 2007 that never got any further than build options and behaviour description despite several requests for more info.  One day somebody is going to have to setup several machines build 3.0 a couple of times with multiple features on or off and try to replicate the described problem.  Then build 3.HEAD and try to replicate X all over again starting with the same tests. Then try to figure out how its happening and whether a fix might be possible.  Well, thanks for the report anyway, at least we know to watch for it meanwhile.

Every few months like today, I trawl through these, just over 100 bugs now, looking for ones which I could maybe replicate in less than a few days. Today I’m slightly happy. One died as invalid, and two died as no longer fixable. The whole area of code they might have been in is dead.

 

… bugs …

The shorted lived bugs by comparison, are reports that version a.b.c. is doing X when it should be doing Y. They include in the case of crashes and segmentation faults a backtrace of the code (with symbol names). They include header or packet traces if the problem was communicating to some other software. Some even came with information tracking down which feature and piece of the code was going wrong. Found, tracked, patched, tested, and fixed in the main code in mere hours of work.  A HUGE thank you to everyone who reports bugs like this.

 

… betas …

Squid-3.2 has started its beta releases this year. Following the fluffy “grand plan” of being a bit more predictable:  branch in Jun with monthly betas through until a stable series probably appears Dec or Jan.

The fluffy part is caused by some more structural cleanup projects we really really want to make the code easier to work with. Well, late Oct and about half of the cleanups have been done. The stable part is today looking more like next Jan/Feb provided the rest are done and working soon.

Another post closer to 3.2 stable release will cover whats going on there in the features. Some cool stuff. If you want a sneak preview the release notes are already available.

 

… and breakages.

Ah, well. Some may recall 3.1.5.1 and friends. The background there for the inquisitive is that two feature changes clashed head-on and killed stability for two months.

I have irregular but available contact with nine of the maintainers packaging Squid for popular systems. It started slowly, growing until half of them had informed me that important sections of their users were being hit by the lack of IPv6 split-stack support in 3.1.

It surprised me a bit how many people and what in situations are insisting that IPv6 be turned off, but only partially. This was a big problem for early 3.1 series since the build process actively probed IPv6 capabilities and turned various capabilities on or off inside Squid. So an essentially simple change was made, moving the active testing into Squid startup process. Squid builds with IPv6 anywhere and runs it when available. No such luck on simple, enter all the little quiet bits assuming IPv4-only or IPv6-only sockets.

The other feature change that made the recovery even messier was a version bump to libtool in our package bundling system. In short a very simple basic upgrade omission on our part broke the tarball package. A veritable storm of hacks appeared to work around the broken tarballs and extended the fix time as testers disappeared with “it works for me now” leaving the systemic problem unresolved.

3.1.8 finally got some pedantic methodical testing before release and a fix for both problems.

 

Background to all this drama we have had an increase in focus on HTTP/1.1 support over the year from The Measurement Factory. Many of the minor behaviour bug fixes made it into the 3.1 series and the last of the big changes being held back to 3.2.

Continuous Integration

For the last few years there has been a slow growing improvement to the testing and QA Squid is subject to. This last week has seen the construction and rollout  of a full-scale build farm to replace some of our simple internal testing. Robert Collins covers the growth process in his blog.

Here is the initial release notice:

Hi, a few of us dev’s have been working on getting a build-test environment up and running. We’re still doing fine tuning on it but the basic facility is working.

We’d love it if users of squid, both individuals and corporates, would consider contributing a test machine to the buildfarm.

The build farm is at http://build.squid-cache.org/ with docs about it at http://wiki.squid-cache.org/BuildFarm.

What we’d like is to have enough machines that are available to run test builds, that we can avoid having last-minute scrambles to fix things at releases.

If you have some spare bandwidth and CPU cycles you can easily volunteer.

We don’t need test slaves to be on all the time – if they aren’t on they won’t run tests, but they will when the come on. We’d prefer machines that are always on over some-times on.

We only do test builds on volunteer machines after a ‘master’ job has passed on the main server. This avoids using resources up when something is clearly busted in the main source code.

Each version of squid we test takes about 150MB on disk when idle, and when a test is going on up to twice that (because of the build test scripts).

We currently test:

• 2.HEAD
• 3.0
• 3.1
• 3.HEAD

I suspect we’ll add 2.7 to that list. So I guess we’ll use abut 750MB of disk if a given slave is testing all those versions.

Hudson, our build test software, can balance out the machines though – if we have two identical platforms they will each get some of the builds to test.

So, if your favorite operating system is not currently represented in the build farm, please let us know – drop a mail here or to noc @ squid-cache.org – we’ll be delighted to hear from you, and it will help ensure that squid is building well on your OS!

-Rob

That just about covers everything. Hardware and build software requirements are listed in the build farm page.

Hi, a few of us dev's have been working on getting a build-test
environment up and running. We're still doing fine tuning on it but the
basic facility is working.

We'd love it if users of squid, both individuals and corporates, would
consider contributing a test machine to the buildfarm.

The build farm is at http://build.squid-cache.org/ with docs about it at
http://wiki.squid-cache.org/BuildFarm.

What we'd like is to have enough machines that are available to run test
builds, that we can avoid having last-minute scrambles to fix things at
releases.

If you have some spare bandwidth and CPU cycles you can easily
volunteer. 

We don't need test slaves to be on all the time - if they aren't on they
won't run tests, but they will when the come on. We'd prefer machines
that are always on over some-times on.

We only do test builds on volunteer machines after a 'master' job has
passed on the main server. This avoids using resources up when something
is clearly busted in the main source code.

Each version of squid we test takes about 150MB on disk when idle, and
when a test is going on up to twice that (because of the build test
scripts).

We currently test
2.HEAD
3.0
3.1
3.HEAD

and I suspect we'll add 2.7 to that list. So I guess we'll use abut
750MB of disk if a given slave is testing all those versions.

Hudson, our build test software, can balance out the machines though -
if we have two identical platforms they will each get some of the builds
to test.

So, if your favorite operating system is not currently represented in
the build farm, please let us know - drop a mail here or to noc @
squid-cache.org - we'll be delighted to hear from you, and it will help
ensure that squid is building well on your OS!

-Rob