Squid’s in the Clouds

January 4, 2015 by

After the incident to the Project’s main server the Sysadmin team has started looking for ways to improve the reliability and performance of all of our services.

Our sponsor Rackspace has donated us a number of virtual machines, and we decided to leverage that for most of our core infrastructure. The main website, official source code repository, bugzilla and other essential services have since been migrated to virtual machines. Some of the benefits we expect to obtain are improved reliability, easier upgrades, simpler backups; we will not be abandoning physical nodes, there are some tasks that can’t be easily virtualized.

Our continuous integration testing farm used to run on a number of smaller VMs, but this was quite inefficient: smaller VMs would spend most of their time idle, and wouldn’t have much horsepower when active, resulting in relatively long build times. We currently test each commit on over a dozen different OSes and OS versions, across three different compilers. Each run requires over four and a half hours.

Docker has gained lots of mindshare in recent months as a lightweight, flexible and functional containerization platform, and after giving it a try I proposed to the project sysadmins to adopt it as the Project’s platform of choice for most of our build farm, on top of a beefier VM.
After a bit of testing, the results are very satisfying. Our main Linux farm infrastructure has changed from 12 single- or dual-core VMs to 1 8-core system running a dozen build-node containers. Build times are down by 25% and we are able to more easily overcommit CPU and memory resources; disk usage is dramatically down: while a full VM typically reqiures 40 GB of HDD space, a container only needs 4, half of which is ccache data.

As Docker containers are not visible to the network unless explicitly configured, security is up and we need less work to maintain them. It would be rather trivial to fully automate the deployment of newer versions of several Linux distributions by writing a few Dockerfiles.

Technically, most of this could be obtained by chroot(2) call and the right setup; we are only scratching the tip of the cloudy containerization iceberg. But being able to migrate a nontrivial setup in only a few hours of work could say that in the right hands the tools are already in a pretty advanced state.

Squid-3.2 mythbusting NAT

December 19, 2014 by

One of the more frequently mentioned “problems” with Squid-3.2 since its release is a change in how it handles NAT failures.

The Myths

“Squid used to work when I NAT traffic to it from my router.”

“Squid used to work with one port when I configure the browser and NAT traffic.”

The Reality

No. Squid up until 3.1 would silence the NAT errors and treat the router as if it were the client browser. Any differences between the Host header and the requested URL were also completely ignored. All of this would be invisible to you the administrator, hidden at debug levels not normally shown. With several of the problems being completely unrecorded as well.

This last part seems to be behind a fair bit of skepticism about whether the problem we solved really exists. Nothing was showing up in the logs before and even a full packet trace from Squid or a gateway server would not reveal how JavaScript hijacks a client browser to pass internal documents to an external attacker.

What Changed?

Squid-3.2 finally added Host header validation to protect against the nasty behaviour and a few 0-day attacks using CVE-2009-0801 security vulnerability. Which has been a known flaw with NAT interception by HTTP proxies for at least 12 years now. This meant two things had to change:

* ignoring NAT errors had to stop. We can’t rely on validation results if the TCP details are already known to be corrupted.

* traffic directly from a browser to a NAT intercept port on Squid had to stop. There is no way to separate the NAT lookup result of error and unknown entry. A pity really, but that is what we are forced to work with.

During testing we uncovered a lot of really quite nasty behaviour by various clients, and indeed some public services setup to take advantage of the NAT bug as if it were a feature. On the whole a lot of client software sends a Host header with values strangely unrelated to what is being requested of the proxy. All of this had to go, or did it? for most of the year or so of testing it was banned completely and a HTTP error message returned for any such garbage. But in the end it was clear that we had to let it through somehow or we would be pitting Squid against the biggest players on the Internet … yeah.

(Almost-) Final Results

Squid-3.2 and later will reject traffic where NAT lookups fail on an intercept port. This includes NAT done on external devices and browsers directly sending proxy requests to the Squid intercept port. At the very least accurate reporting about the traffic and what it contains is the critical factor. If we let this traffic into Squid it would seriously compromise the validation results and also allow malicious clients to attack internal network resources with a large measure of anonymity. Neither of which is acceptible for a proxy trusted with controlling user traffic.

The best practice guideline for some years has been that NAT MUST be done on the Squid device. Squid-3.2 are now enforcing it as a basic requirement. If you are one of the network administrators running into this requirement change, please investigate the Policy Routing functionality of the device you originally had doing the NAT.

Squid-3.2 and later will accept invalid Host headers and produce a response to the client. But they will not cache the untrustworthy transaction results. They will contact the same server which the client TCP connection would have originally reached had Squid not been there (visible as ORIGINAL_DST in the logs). Notice that this is now deserving of the name “transparent proxy” a lot more than previous intercepting Squid. Even so NAT interception is still only half-transparent with the server being able to easily identify the proxies existence.

The loss of caching is intended to be a temporary solution until we can properly implement per-client caching of objects. As the saying goes there is nothing so permanent as the temporary – Squid still contains this workaround several series later. Support with this work is very welcome.

The situation when upstream peers are involved is also quite dangerous. For now we have had to permit Squid to pass the traffic to peers, which opens all multi-hop systems of proxies to the same vulnerabilities that were previously possible wth a single intercepting proxy. The solution is also going to require a substantial amount more work and be some years away. For the meanwhile it is a good idea to avoid passing intercepted traffic to cache_peer when possible.

More information on the problems, log entries, when they occur and what can be done about each is detailed in the Squid wiki Host header forgery page.

squid-cache.org outage

September 7, 2014 by

We are currently weathering both a PSU meltdown and disk failure (with full compliment of side effects) on the main squid-cache.org server. The Squid sysadmin and Foundation are all working on it as fast as possible.

Donations towards the purchase of a new server are greatly appreciated and will allow us to buy better hardware.

UPDATE: 2014-10-08: after weeks of late nights and very early mornings by the volunteer sysadmin team we are nearly all back up to full operational state again. The latest status of each affected major service is outlined below.

Mail and Mailing lists

The mail server for squid-cache.org was worst hit. Email has been down across most of September since the initial event. Any email sent to a squid-cache.org email address will have been held up and/or bounced.

Mail services are now back, but some spam control upgrades were forced on us that are still having fallout. Mailing lists are being migrated to a new domain name lists.squid-cache.org. Existing subscriptions have been automatically moved to the new list domain. You can expect to see an initial post explaining the change when the list you are subscribed to is recovered.

This change may require updates to mail filtering and rules outside our services. If you are aware of any in your domain or systems please see that they are updated.

IMPORTANT: some subscriptions have subsequently had to be removed due to backscatter spam from relays and corporate ticket logging systems. Posters to the list know who I am talking about. If you find your subscription has gone silent again recently please check the systems you are having mail delivered to and through then try re-subscribing to the new list.

Mail archives are currently split between the old hypermail + swift archival system and new pipermail. This is reflected on the website. If you are maintaining a mirror of the Squid mail archives please subscribe to our new mailing list for mirror operators and get in touch with the sysadmin team to sort out what is going to happen with mail mirrors in future.


We believe this is recovered. If anyone is still having issues resolving the domains please get in touch with noc @ lists.squid-cache.org.

Code Repository

The repository has been fully recovered and service on bzr.squid-cache.org and rsync is resumed.


The FTP service has been limping along with access but no updates. The main server is now in the process of being rebuilt from scratch. Please do not be surprised if you are suddenly challenged for login, try a mirror instead. Anonymous access to the main FTP will be resumed ASAP.


The http://www.squid-cache.org site is mostly up and running. Mirrors have remained available for the duration, but were not being updated with daily contents. The updates should now have resumed, but there are still a few kinks to work out in the content. If you find any issues going forward please report it in our bugzilla under Project Services.

Mirror Services  and rsync

If you are running a WWW mirror please ensure you are using rsync access and your server is capable of serving the http://www.squid-cache.org name as outlined in the mirror guidelines. Similar goes for FTP mirrors. We are adding a new mailing list for mirror server contacts. Our database of registered contacts for HTTP and FTP mirrors will be automatically subscribed so please keep an eye on the mailbox you registered with us already. Anyone running a Squid mirror of any kind please subscribe and post your mirror details to the list.

The rsync service itself is running with some data shares temporarily disabled. These will be re-opened as the services are brought back to full functionality. There are no changes to remote configurations provided you have been following the current mirror guidelines. The dynamic website (http-files-dyn) will no longer be publicly available, please mirror the static (http-files) instead.

Apologies for the inconvenience.

… and Murphys Law has not finished with us yet:

Some security vulnerabilities were reported. A new squid-3.4.8 package has been released to resolve those. All users relying on SNMP or the pinger helper are advised to upgrade. The SNMP details can be found here, pinger details can be found here.

HTTP/1.1 update obsoleting RFC2616, is complete

June 7, 2014 by

If you have not been aware of the IETF HTTPbis Working Group and what we do, it is chartered to improve HTTP. For the last decade and a half  HTTP/1.1 has been defined by the monolithic and sometimes confusing RFC2616 document with a relatively few extensions. The WG has been putting in a lot of effort to simplify the texts and clarify how the protocol actually works.

If you have been putting off reading the HTTP/1.1 specification because of its enormous length now is a good time to dive in. The text has never been simper and easier to read. Changes from the old document have been kept minimal, but there are some listed in the Appendices.

Mark Nottingham the WG chairman made this formal announcement a few hours ago:

The revision of HTTP/1.1’s specification, obsoleting RFC2616, is complete.

 http://tools.ietf.org/html/rfc7230 – Message Syntax and Routing
 http://tools.ietf.org/html/rfc7231 – Semantics and Content
 http://tools.ietf.org/html/rfc7232 – Conditional Requests
 http://tools.ietf.org/html/rfc7233 – Range Requests
 http://tools.ietf.org/html/rfc7234 – Caching
 http://tools.ietf.org/html/rfc7235 – Authentication

Along with the related documents:
 http://tools.ietf.org/html/rfc7236 – Authentication Scheme Registrations
 http://tools.ietf.org/html/rfc7237 – Method Registrations

Thanks to everyone who has commented upon, reviewed and otherwise contributed to them over this nearly seven-year(!) effort.

Special thanks to our Area Directors over the years: Lisa Dusseault, Alexey Melnikov, Peter Saint-Andre and Barry Leiba, along with Yves Lafon, who helped edit Range Requests.

Finally, please warmly thank both Roy Fielding and Julian Reschke the next time you see them (I believe beer would be appreciated); the amount of effort that they put into these documents is far, far more than they originally signed up for, and they’ve done an excellent job.

Now, onwards to HTTP/2

P.S. This document set’s completion also has enabled the publication of these related non-WG documents:
 http://tools.ietf.org/html/rfc7238 – The Hypertext Transfer Protocol Status Code 308 (Permanent Redirect)
 http://tools.ietf.org/html/rfc7239 – Forwarded HTTP Extension
 http://tools.ietf.org/html/rfc7240 – Prefer Header for HTTP


Oh! And one more thank you, to Mark Baker for serving as Shepherd for the Caching doc.


Zero-Sized Replies from Windows Servers

April 30, 2014 by

During the last few months there have again been a number of bug reports and queries from administrators seeing Zero Sized Reply error pages being produced by Squid 3.2 and later.

These “errors” are produced when Squid sends an HTTP request, then something out in the network goes wrong and the TCP connection gets severed while Squid is still waiting for the start of HTTP response to arrive. As you can imagine this is a little vague because that “something” is any one of a large set of potential networking problems.

Investigation of the old usual culprits in ECN, Window Scaling, PMTUd, and CONNECT proxying ruled them out leaving us mostly in the dark.

Testing without the proxy appeared to work fine. As did small short transactions even through the proxy. Leaving us more than a little confused.

The most common theme this time seems to be Windows based SSL/TLS services with recent but not top of the line software versions. IIS or Sharepoint on Server 2008 and 2010 for example.

Daniel Beschorner has done some investigating and reported this:

Since Squid 3.2 the SSL flag SSL_OP_ALL is no longer enabled by default in Squid. It enables different workarounds in the OpenSSL library.

Windows / IIS seems to get confused by empty packets (to mitigate the BEAST attack) sent from OpenSSL in TLS 1.0.

So the possibilities are:

We have also had remarkably similar problem reports about iTunes servers. That one is still unconfirmed and unresolved.

Welcoming a Squid Project sponsor: Rackspace

November 14, 2013 by

Testing is a very important part of the life of a project such as Squid, especially as we wish to support as many different operating systems and platforms as we can, as efficiently as we can.
This also means that we can’t restrict to well-known POSIX APIs, but we need to account for each OS’s particular quirks and optimizations, especially in those areas where there is the need of a deep integration with the OS.

One of the tools which we have added a few years ago is the build farm. When new code is committed to a followed branch, our Jenkins setup will notice it, and schedule a test run on all the eligible nodes. This means we need to rely on a mix of operating systems, libraries and compilers.

We are always on the lookout for sponsors, both individuals and companies, and back in July Rackspace volunteered to donate space for the Squid project us on their public cloud infrastructure. As of now, we have 11 virtual machines and one database hosted in their cloud, running mostly various flavors of Linux and FreeBSD. The experience has been very satisfying, in particular I’ve found the claim about “fanatical support” to be well-founded in the dedication and kindness of everyone who was involved in making this happen.

It was not all rose petals, of course. Our needs mean that we are better off with something resembling the bare metal, while the model Rackspace offers is more oriented towards having predefined images which configure certain aspects such as network addresses, admin users’ passwords etc.

Still, it is very refreshing that if an OS is among the list of supported images (as an example, Ubuntu Saucy was available about three days after Canonical released it) setting up a new build node in the farm takes literally 15 minutes, most of which is spent waiting for the server image to be cloned from the template.

The first effects of the newly expanded build farm are already apparent: we have reached a stable build status on a wide number of Linux, FreeBSD and OpenBSD versions, with gcc, clang and icc compilers. We are starting to set sight on MacOSX and to bring Squid 3 to Windows. All of this would have been much harder, or even impossible, without the Build Farm.

So on behalf of everyone who works on Squid, I’d like to thank Rackspace for their kind contribution to the Project.

Squid Software Foundation Board of Directors Position Vacancy

September 10, 2013 by

The Squid Software Foundation is seeking to expand the board of directors. We currently have three directors and looking for at least one more to join the team. For details about the position and what the directors do please see http://www.squid-cache.org/Foundation/director.html

Being a Squid Software Foundation Director is a serious responsibility, but also a cool gig! Not only can you have an immediate and significant impact on the Squid Project, but you can earn admiration and respect of your peers while doing more than just your usual software development, system administration, or support activities.

Do you want to brag about being more than a successful geek? Exercise the parts of your brain you did not know you had? Resolve real-world conflicts and balance real-world trade-offs? Then how about solving a few difficult Squid Project problems? Want to spice up your resume or simply learn to manage a popular open source project? Consider nominating yourself!

Applicants should contact board@squid-cache.org with nomination for the position of Director. Self-nominations are accepted and encouraged. Please indicate why you think the nominee would be a good Foundation director.

Please submit nominations by October 4th, 2013.
The Squid Software Foundation Board of Directors
Henrik Nordström,
Amos Jeffries,
Alex Rousskov.

The Squid Software Foundation

July 17, 2013 by

As many of you know by now, the Squid Project has formed the Squid Software Foundation, a non-profit organization to support Squid developer and user communities. The Foundation aims to provide community governance and representation; logistical and administrative assistance; infrastructure resources; as well as copyright ownership and licensing authority for the Squid Project.


The Foundation has been recognized by US government as a 501(c)3 public charity. This recognition is meant to assure the public of our not-for-profit status, exempt Foundation from paying most taxes, and to make qualified donations to the Foundation tax-deductible. This designation is a big deal, and it took a lot of effort to get this status. A big “Thank You” to those who made it possible!

The Foundation structure has been modeled after the FreeBSD Foundation, with various bits and pieces borrowed from the Apache Foundation and other well-run open source non-profits. You can find our bylaws and other documents at


The Foundation is currently being run by three volunteers: Henrik Nordström, Amos Jeffries, and Alex Rousskov. There is a lot more work than the three of us can handle, and there are plans to expand the Board of Directors to at least four directors this year. Watch this space for further announcements on the subject.

While the Foundation is not going to participate in Squid development, there are plenty of infrastructure and community activities where we can help both Squid developers and administrators. The scope of possible projects ranges from release maintenance, to paid services coordination, to build farm administration, to automated regression testing, to experimental live deployments. With sufficient funding, we may even be able to organize developer meetups and user workshops.

The Foundation is funded by donations. The startup capital was provided by your PayPal payments that the Squid Project informally accumulated since 2007 and a Measurement Factory contribution. We will need a lot more funds to fulfill the Foundation mission though :-). Donations are now gratefully accepted at


If you have questions about the Foundation, please email them to


Thank you,

The Squid Software Foundation Board of Directors
Henrik Nordström,
Amos Jeffries,
Alex Rousskov.

Squid-3.2: managing dynamic helpers

May 2, 2013 by

One of the new features brought in with Squid-3.2 is dynamic helpers. A brief name for a very useful administrative tool and like all tools can be both easy and tricky to use at the same time.

If you have a proxy using helper processes but only a small cache (or none) this is a feature for you.

The good news

Configuration is super easy – just set the initial startup, maximum number of helpers and an idle value for how many to start when new ones are needed to handle the request load.

Dying helpers have a higher threshold before they kill Squid. It is not perfectly tuned yet in the code so improvements will contnue to happen here, but already we see small bursts of helper failures being suppressed by re-started replacements without that all too familiar Squid halt with “helper dying too quickly” messages. Note that this is just higher, not gone.

The bad news

Determining what those values should be is no more easy or straightforward than before. Squid uses fork() to start new helpers. The main side effect of this is that helper instances started while Squid is running will require a virtual memory size equivalent to the Squid worker process memory at the time they are started. If your proxy is pushing the box to its limit on RAM, dynamically started helpers could easily push it over to swapping memory at the worst possible time (peak load starting to arrive). Also on the bad news side is that the helpers are run per-worker. Which has the potential to compound the RAM usage problems.

We do have a proposal put to the development team which will almost completely remove this problem. Having the coordinator or a special spawner kid do the forking instead of the heavy workers. But as of this writing nobody is working on it (volunteers welcome, please contact the squid-dev mailing list).

Practice Guidelines

While it may look like the bad news is worse than the good news it does turn out that most installations are small instances or non-caching worker proxies these days. All of which may need lots of helpers, but are not heavy on the RAM requirements. For all these installations dynamic helpers are ideal and in a lot of cases can even be set with zero helpers on startup for a very speedy delay to first request accepted time.

The caching proxy installations with higher memory requirements in the workers can still make use of the dynamic nature to avoid complete outages in the worst-case situations where peak traffic overloads the planned helpers. But should normally be configured as before with enough helpers to meet most needs started before the RAM requirements become too onerous on the worker.

Until at least the bad news problems above are resolved the default behaviour for Squid will continue to be starting all the maximum helpers on startup. So there are no unexpected surprises for upgrading, and the old advice on calculating helper requirements is still useful for determining that maximum.

Squid-3.2: Pragma, Cache-Control, no-cache versus storage

October 16, 2012 by

The no-cache setting in HTTP has always been a misunderstood beastie. The instinctual reaction for developers everywhere is to believe that it prevents caching or cache handling or some such myth.

This is not true.

By definition it merely forces caches to revalidate existing content before use (ie it tells the proxy to “be ultra, super-duper conservative. Do not send anything from cache without first contacting the server to double check it.”).

When sent on a client (browser) request:

  • Pragma:no-cache instructs HTTP/1.0 caches to revalidate any cached response before using it.
  • Cache-Control:no-cache instructs HTTP/1.1 caches to revalidate any cached response before using it.
  • Pragma:no-cache only works for HTTP/1.1 caches when Cache-Control is missing.
  • all other values of Pragma are undefined and to be ignored.

When sent on a server response:

  • Pragma in all its forms has no meaning whatsoever. It must be ignored.
  • Cache-Control:no-cache instructs HTTP/1.1 caches to revalidate this response every time it is re-used.

If you read those bullet points above very carefully you will notice that at no point is store mentioned. None whatsoever. The closest it gets is mentioning what to do with already-stored content (revalidate it). In fact the HTTP/1.1 specification goes as far as to say explicitly that responses with no-cache MAY be stored – provided the revalidation is done as above.

no-cache in Squid

The well-known squid versions of the past have all been HTTP/1.0 compliant and advertised themselves as HTTP/1.0 software. These proxies both looked for Pragma:no-cache headers and obeyed them:

  • Squid being HTTP/1.0 that Pragma took precedence over Cache-Control.
  • Due to lack of full HTTP/1.1 revalidation in very old versions Squid has traditionally treated no-cache in either header as if it were Cache-Control:no-store.
  • Due to some old server software Pragma:no-cache on responses was treated as a mistaken form of Cache-Control:no-store.

Starting with version 3.2 Squid is advertising and attempting to fully support HTTP/1.1 specifications. This is a game changer.

All of the above is about to be up-ended, assumptions can be thrown away and some funky cool proxy behaviour allowed to take place.

Hiding in the background is the instruction that Pragma only applies when Cache-Control is missing from a request. We can ignore it – almost completely. When we do have to pay attention we only need to notice the no-cache value and can treat it as if we received Cache-Control:no-cache.

The other change is a potential game changer: The object being transfered is stored now, revalidated later.

Some implications from storing no-cache responses:

  • servers can utilize 304 responses instead of generating new content. Saving a lot of bandwidth and CPU cycles.
  • all those configuration hacks for ignoring or stripping no-cache are no longer needed. Also, the harm they do will become more visible as revalidation is skipped.
  • cache HIT ratio can potentially rise above 50% for forward proxies. As a side effect of the HIT counting market a large portion of web traffic is utilizing no-cache instead of no-store or private. This large portion is cacheable but until now Squid has been dropping it.

Before the marketing department panics about the end of the world lets be clear on one important point:

revalidation means every client request will still reach the end server doing HIT counting, traffic control, whatever – but in a way which allows 304 bandwidth optimization on the responses.

Do not expect a sudden rise of TCP_HIT in the proxy logs though. It is more likely to show up as TCP_REFRESH_HIT or the nasty TCP_REFRESH_MODIFIED/TCP_REFRESH_MISS which is produced by broken web applications always sending out new unchanged content.


Get every new post delivered to your Inbox.

Join 39 other followers