From 3.1 Squid now supports Automatic Language Negotiation.  There seems to be a little bit of confusion over what this means and what should be configured.

Obviously we would like people to enable and use the automatics. For some very good reasons which you shall understand at the end of this post. I would hope you agree by then too.

Most software you and the rest of the world will be familiar with comes in two  forms: English, or translated into your own language. You might have your computer set to  non-English language and all the software that can changes text so you can more easily read it.

All of this is very you-centric and only affects whatever machine you are using. The www is a very different beast altogether. It has to deal with everyone. At the same time too.

The best example is search engine results. You may have noticed when you do a search that some results have little tags. cached, similar pages, more, … and sometimes one called ‘translate’.  This is nice, because it means the search engine has noticed that the page is in a language you may not know and its offering a link that will translate the page to one you can read.

Ever wondered ‘how does it know’? and more importantly;  what does all this have to do with Squid?

Lets start with the second one:  What does this have to do with Squid?  well Squid. The one I run, the one you probably run, and many others around the world generate error pages.  You are sure to have seen the “404 Not Found” at some point. Probably “Access Denied” and “Connection Failed” as well.

Until now Squid has been setup and managed by someone for a specific purpose. That person sets the language those pages are displaying to something they can read and see what problems are. And here is where the confusion seems to start.

One admin who setup the new Squid promptly changed the error_directory language to German (de). Quite rightly he thought. I’m German, my customers are German, who needs any other languages installed? It will only confuse me to see other language errors. And the server is set to German so it won’t show any others anyway.

At this point I’m guessing you might agree with some or all of that assumption. For your language in the same situation, you would probably do the same yes?

Lets take a look at that search engine question. We found a website. It is written strangely in Persian. We do not have a clue whats its about. Clicking on the ‘translate’ link and we read the page.

But wait, …

… we only saw one single ‘translate’ link and surely the engine knows many languages. We should see a whole bunch, one for every language the page might be translated into.

This is where we get closer to Squid again. The HTTP protocol has a header where the browser says what languages its current user would like things displayed in. The search engine is reading that header and only showing the translate link for most prefered language it can cope with.

This is precisely what Squid now does for the error pages it creates. The language displayed depends on the visitor doing the reading when the automatics are allowed to run.  The server Squid runs on has nothing to do with the language.

Our German admin if you recall set the error_directory to German so he could read it.

Too bad for us if you or I non-German readers had a problem getting to one of his customers websites. Or if we were visiting one of his customers and using their Internet access from our laptop.

What he should have done was leave error_directory unset. When he visits the proxy to test a problem it shows german language, because has browser says to. The user who reported the problem might be reading the same message in Chinese, or Korean.

Squid provides error pages for two reasons, to explain whats gone wrong, and to explain to someone what to do about the problem.  In this world of many international people your visitors and users could be coming from any kind of background with any kind of language needs. To help reduce the number of strange language half-understood complaints we all receive the Squid team have made Squid explain things in a language the visitor can read, so you don’t have to. All you have to do is turn it on.

http://wiki.squid-cache.org/Translations#What_has_been_done.3F

Squid now speaks in over 130 national languages and dialects. 100 more than this same time just last year. Some are more complete than others, improving all the time.

Kia Ora koe.

For the last few years there has been a slow growing improvement to the testing and QA Squid is subject to. This last week has seen the construction and rollout  of a full-scale build farm to replace some of our simple internal testing. Robert Collins covers the growth process in his blog.

Here is the initial release notice:

Hi, a few of us dev’s have been working on getting a build-test environment up and running. We’re still doing fine tuning on it but the basic facility is working.

We’d love it if users of squid, both individuals and corporates, would consider contributing a test machine to the buildfarm.

The build farm is at http://build.squid-cache.org/ with docs about it at http://wiki.squid-cache.org/BuildFarm.

What we’d like is to have enough machines that are available to run test builds, that we can avoid having last-minute scrambles to fix things at releases.

If you have some spare bandwidth and CPU cycles you can easily volunteer.

We don’t need test slaves to be on all the time – if they aren’t on they won’t run tests, but they will when the come on. We’d prefer machines that are always on over some-times on.

We only do test builds on volunteer machines after a ‘master’ job has passed on the main server. This avoids using resources up when something is clearly busted in the main source code.

Each version of squid we test takes about 150MB on disk when idle, and when a test is going on up to twice that (because of the build test scripts).

We currently test:

• 2.HEAD
• 3.0
• 3.1
• 3.HEAD

I suspect we’ll add 2.7 to that list. So I guess we’ll use abut 750MB of disk if a given slave is testing all those versions.

Hudson, our build test software, can balance out the machines though – if we have two identical platforms they will each get some of the builds to test.

So, if your favorite operating system is not currently represented in the build farm, please let us know – drop a mail here or to noc @ squid-cache.org – we’ll be delighted to hear from you, and it will help ensure that squid is building well on your OS!

-Rob

That just about covers everything. Hardware and build software requirements are listed in the build farm page.

Hi, a few of us dev's have been working on getting a build-test
environment up and running. We're still doing fine tuning on it but the
basic facility is working.

We'd love it if users of squid, both individuals and corporates, would
consider contributing a test machine to the buildfarm.

The build farm is at http://build.squid-cache.org/ with docs about it at
http://wiki.squid-cache.org/BuildFarm.

What we'd like is to have enough machines that are available to run test
builds, that we can avoid having last-minute scrambles to fix things at
releases.

If you have some spare bandwidth and CPU cycles you can easily
volunteer. 

We don't need test slaves to be on all the time - if they aren't on they
won't run tests, but they will when the come on. We'd prefer machines
that are always on over some-times on.

We only do test builds on volunteer machines after a 'master' job has
passed on the main server. This avoids using resources up when something
is clearly busted in the main source code.

Each version of squid we test takes about 150MB on disk when idle, and
when a test is going on up to twice that (because of the build test
scripts).

We currently test
2.HEAD
3.0
3.1
3.HEAD

and I suspect we'll add 2.7 to that list. So I guess we'll use abut
750MB of disk if a given slave is testing all those versions.

Hudson, our build test software, can balance out the machines though -
if we have two identical platforms they will each get some of the builds
to test.

So, if your favorite operating system is not currently represented in
the build farm, please let us know - drop a mail here or to noc @
squid-cache.org - we'll be delighted to hear from you, and it will help
ensure that squid is building well on your OS!

-Rob

From early inception when the developers have nothing but dreams for it.  Through the coding and arguments about what should be included and how. Through the alpha testing with its harrowing hours pondering obscure code from last decade. Even the odd period of panic as security bugs are whispered about behind closed doors. Such is the early life of software.

Two weeks ago word went out that 3.1 was reaching end-game.

This part of the release lifecycle seems to be going well. Packages appearing very slowly as QA throws demanding eyes on the code and making us actually fix things. Don’t be fooled by the packages out already, they have been in QA for a few months to get this far. On that note:

NetBSD, Gentoo, Ubuntu, FreeBSD and RedHat already have packages ready and available for at least testing use if you know where to look (ie the links right there might be a good start).

Debian has a bit more QA to go as of the writing, but the maintainer tells me there will be packages out soon.

OpenBSD and Mac turned out at the last minute to be running split-stack IPv6 implementations (for security apparently). All the documentation read in two years left the impression it was a Windows XP anarchism (and who runs XP Pro on a server?), so support was delayed and delayed.  The OpenBSD maintainer and someone interested from Mac are working with myself on closing that gap in the features.

There may be more OS with 3.1 packages. I’ve only begun working my way down the distrowatch.org popularity list to see which OS do and who to contact. Squid has bundles on over 600 OS apparently.

If you know who does the official packaging for your OS and whether there are 3.1 packages ready, please do me a favor and mention it. I’m seeking a web page where to find the squid (or squid3/squid30/squid31) package information and also the place where distro bug reports about Squid might end up.

One of the major changes we made to 3.1 during its life cycle was the introduction of first-past-the-post feature additions instead of pre-planned and sealed roadmaps. Combined with a regular short release timeline instead of indeterminate long testing periods.

So why has 3.1 taken more beta releases than 3.0?

Unfortunately before that policy was implemented we did go down the old track a fair way and guaranteed that several features would make it into 3.1.  One of these (SourceLayout) has proven to be a lot of slow work. (Not major re-writing thankfully).

The other new features code for 3.1 have been tested and stabilized between 3.1.0.1 and 3.1.0.6 and are happily in use on many production environments.  But this last remaining feature still has a lot to go in and be tested for bugs. The result is 6 months delay on 3.1 and a handful of extra candidate releases, even some blockage of new stuff ready to go into 3.HEAD for the next release.

3.2 is starting off without this old legacy blocking problem, so it should be the first release to really gain the benefit of a much faster testing and release stages of production.

There are also a bunch of bugs which seem to be important but we are unable to track down and fix yet. Mostly holdovers from 3.0. The basic policy is to block release until all open bugs major or higher are closed. We might downgrade bugs to normal though if they are voted to be ignored.

While we wait for the last feature to stabilize some of us still try to work away at that list, additional volunteers are very welcome to speed things up.  Meanwhile I would like some opinions please about the final stage for 3.1:

Kinkie pointed out Linus Torvalds blog today to the rest of us here working on Squid. As the release maintainer for Squid-3 this year I kind of agree, its a sad time to cutting a new version. For me its more of a reflection that for all the high hopes we have of this new release, we had the same or similar hopes of the earlier one. Just 12 months ago now.

On that sad note, yes its finally happened. 3.0 has aged into a full blown stable package. Most of a month and no new bugs. Perfect time for something shiny and new for the neo-tech fanclub. And so with that for an intro we are gone for 3.1 !

3.1 is available for beta testing in the form of 3.1.0.1. see the Release Notes for further details on the finer details of change.

This release has gained from the experiences of 3.0 and 2.6, starting from a much more stable base of code than the initial. 3.0 had a long period of years with few active developers, an interminably long period of testing releases, and in hindsight a premature birth.

Alongside the code this release has a wider collaboration with active users. For the first time in many years we held a Developer meeting that included Users. We who were there certainly took in a lot of feedback from all sides. I hope those users who talked to us can see in this release that their comments, even those made in passing, have been listened to and worked on.

The small comment from one user when asked what their biggest itch with squid was “we don’t like these being called STABLE, when its obvious they are not.” has led to the most notable change made to 3.1.  That comment and similar feelings by others lead us into discussions on the release naming and numbering. From which we have produced – 3.1.0.1 – the second milestone point of the branch we are calling 3.1. Where the developers have everything done and working for us.

no more DEVEL, PRE, or RC, no more premature labels guessing when things might be STABLE.  Just 3.1.0.1. Further testing from the rest of you will show whether anyone can consider it stable, unstable, usable or as buggy as raw earth.

From the developers; We use it. We love it. Try it, and see for yourselves.

Some of the stuff you will find there is;

• a lot of small changes aimed towards easier use and configuration (three cheers to those who nagged long an hard for this).
• a lot of network RFC compliance extensions, making 3.1 much more capable of meeting modern network needs. The future still holds improvements, but 3.1 is definitely better in many respects than everything that came before.
• a lot of things to make Squid a better experience for your own users. More seamless network recovery tricks than ever before. We have even tagged along behind the international localization bandwagon in our own way to make the errors squid does have to show both pretty and readable.

Sadly, careful readers will notice a section of the Release Notes labeled “Regressions against 2.7″.  Yes, those of you who moved to 2.7 because you needed some brand new feature there may still have trouble migrating up to 3.1. What we have done is to port as many of the 2.6 features and fixes as we could. A few did not make it in time, but will be coming in 3.2, alongside the features added as experimental in 2.7.

On the overview:

• 2.5 has disappeared over the horizon into the long dark night of obsoletion.
• 2.6 is itself officially aging out now. Supported, but the developer first response is “can you try something newer?”.
• 2.7 is being maintained for the few extremely high-performance accelerator setups. But in general the Squid-2 sequence is aging out for us developers.
• 3.0 has reached a point of stability, though not fully-featured.
• 3.1 is available for testing as the next step up. You should be planning to migrate up to 3.1 or later release.

If there are any features holding you to Squid-2, or even an issues you find with testing Squid-3 speak up, we rely on your input to choose the most needed features for porting.

Thank you all, and enjoy your use of Squid 3.1

We have been getting a growing number of reports and bugs from people using Squid 3.0 described as ’squid producing a blank page’ when bypassing squid apparently works.

Sounds familiar to some yes? I’m bringing it up now because while it is an old problem, its not the TCP issues Adrian wrote about earlier and you should also check if you find its not this. Which incidentally can have exactly the same visible effects for end-users.

This ‘new’ issue is caused by certain widely-used web servers which shall remain nameless and unadvertised by me. Which always respond with HTTP/1.1 chunked-encoding of pages.

Servers are explicitly forbidden from sending that particular encoding type to software announcing itself as HTTP/1.0 (such as squid). But the broken server is doing it anyway!

Ironically: The authors use this server on their own help and support website. So those who are having this problem both see it as a squid problem, and can’t find or see any solution they may have posted anyway.

How to tell if this is your problem?

Use squidclient to make a web request that bypasses the squid proxy. It should send out the HTTP/1.0 request and get a page back. If the headers of the response include “Transfer-Encoding: chunked” there is your problem.

This is currently only an issue in Squid 2.5 or earlier and 3.0, which is still highly modeled around 2.5.

The solutions are varied depending on your capabilities.

Simplest for some will be to just bypass squid for those domains.

[ UPDATE: (thanks Michael Graham)

Apparently several people are having success with simply dropping the Accept-Encoding header to certain of these broken servers. Adding this to their squid.conf :

# Fix broken sites by removing Accept-Encoding header
acl broken dstdomain ...
request_header_access Accept-Encoding deny broken

NP: don't forget to remove it again when you upgrade out of 3.0

]

Next best is to use peer-routing to divert those domain requests at a squid 2.6 (or if you are feeling experimental a 3.1 build)

If its a serious issue and you are accelerating for one of these broken web servers. Then you will need to stick with Squid 2.6 until 3.1 is available for production use.

Why does it work for 2.6 and 3.1 but not 3.0?

Well, things are a bit messy I’ll have to write it up one day. Suffice to say that 3.1 has a lot more HTTP/1.1 support where the chunked-encoding/decoding was intended for. But 2.6 needed it a bit earlier so a version of the decoding (only!) was done to fit 2.6 needs at solving this same issue for high-performance users earlier last year.

The 3.0 code is just different enough that it would need a whole new back-port project to get it going well. The time and work that would take is being used instead to get 3.1 out faster. Which should be within a month of this writing so procrastinating could solve the problem for you.

[UPDATE: Thanks to the Gentoo Project for their work back-porting this will be available from 3.0.STABLE16-RC1 ]

Jason Healy posted some useful information to the squid-users list a week or so ago.

Quoting:

I’ve been a happy user of Squid for the past 10 years or so, and I’d like to take a second to thank everyone who has worked so hard to make such a great piece of software!  I’d like to give back to the Squid community, but unfortunately I’m not much of a C hacker.  However, I’m hoping I can still help.

I’ve just spent a few days getting my school’s Squid install up to date (we were running 2.5 on Debian Woody).  I switched to using tproxy this time around (we used to do policy routing on our core, but it was spiking the CPU too much).  Thanks to the mailing list, some articles on the web, and a little messing around I was able to get the whole system up and running.  I’ve documented the steps here:

http://web.suffieldacademy.org/ils/netadmin/docs/software/squid/

The document is written for someone with a decent grasp of Linux, and is specifically geared to Debian Etch.  There are some tweaks that are pecific to our install (compile-time flags, mostly), but otherwise it’s pretty generic.  Hopefully, this will help someone else out who’s trying to build a similar system, so I’m posting so it will hit the archives.

A few people have asked me what the deal is with Squid-2 and Squid-3.

“Why are you developing on Squid-2 when Squid-3 is now out?”

“Should I upgrade to Squid-3 now that its released?”

I’m focusing on Squid-2 for a few reasons, namely:

• Its what people running high-traffic sites are currently running, and Squid-3 doesn’t work at all for them;
• I was fed up waiting for Squid-3 to be released and for it to become mature enough for users to migrate to before I started my performance work. I gave up about 12 months ago and began planning out the work thats currently going on.
• I’m personally much more familiar with the Squid-2 codebase than the Squid-3 codebase.

So what exactly am I doing to Squid-2? Well, I’m doing all the things to Squid-2 which I personally believe we should’ve done in the C++ Squid-3 branch before all the “new stuff” was added. You can find it all at http://devel.squid-cache.org/changesets/squid/s27_adri.html . A summary of what I’m doing in this first round:

• I’m taking a very sharp scalpel to the codebase and removing all of the extra data copies and buffering which is going on;
• I’m reworking the buffer management so arbitrary sized data buffers can be used, rather than fixed 4k buffers for network/disk traffic;
• I’m reworking the Strings interface to use reference counting and reference underlying buffers, saving on memcpy() and malloc() calls, cutting down on the amount of transient memory used to handle requests and dropping the CPU and memory bus utilisation quite dramatically;
• I’m reworking the dataflow between server->store and store->client to use the above reference counted buffers, so data isn’t memcpy()’ed between layers, again dropping CPU and memory bus utilisation;
• And I’m going to break out as much of the code into external libraries with well-understood dependencies, as preparation for documentation, unit testing and further profiling.

My aim is to fix whatever bugs show up in Squid-2.7 and then in Squid-2.HEAD (which has some of the above included already.) I’ll then start bringing across my changes as they’ve been tested and been found stable. My aim is to have the bulk of the above done within the next month or so and get it into Squid-2.HEAD and concentrate on making it stable before I continue tidying up the dataflow and restructuring the ugly bits of code.

Whats this mean for Squid-3? The Squid-3 guys are doing some great work with things such as ICAP and IPv6 and I hope that they’ll gain more experience with their codebase over the next 12 months or so. I’m certainly not bringing ICAP support into Squid-2 until I’ve reworked the dataflow and tidied up the code enough for ICAP to sit comfortably in the data pipeline, rather than have it bolted onto the side and hooking into strange places where it shouldn’t. (I may bring in IPv6 into Squid-2 soon though!)

Hopefully my work and their work will culminate with the development of the next Squid major version over the next 12 to 24 months. There’s a long way to go though and my main aim here is to get faster, better and shinier code out to the majority of Squid users now so they can benefit from the development, rather than repeating the 4-odd year gap between Squid-2.5 and Squid-2.6. Users hated that.

So whats it mean for you?

• If you want to try out Squid-3; if you want supported ICAP services then try it out.
• Squid-2.X will continue being developed over the next 12 months as time permits, so don’t feel like you have to move to Squid-3.
• If you feel adventurous, try out Squid-2.7. Initial reports are that its stable and slightly less CPU intensive.
• Squid-2.7 is the first version to include changes to allow Youtube and Microsoft Updates caching. It doesn’t do it out of the box, but the support is there, and I’ll be publishing test rules soon to let people start caching this stuff.
• If you feel really adventurous then try out Squid-2.HEAD and report back if you have any issues. It should be even less CPU intensive, but only under certain workloads.