Chunked Decoding

by

We have been getting a growing number of reports and bugs from people using Squid 3.0 described as ‘squid producing a blank page’ when bypassing squid apparently works.

Sounds familiar to some yes? I’m bringing it up now because while it is an old problem, its not the TCP issues Adrian wrote about earlier and you should also check if you find its not this. Which incidentally can have exactly the same visible effects for end-users.

This ‘new’ issue is caused by certain widely-used web servers which shall remain nameless and unadvertised by me. Which always respond with HTTP/1.1 chunked-encoding of pages.

Servers are explicitly forbidden from sending that particular encoding type to software announcing itself as HTTP/1.0 (such as squid). But the broken server is doing it anyway!

Ironically: The authors use this server on their own help and support website. So those who are having this problem both see it as a squid problem, and can’t find or see any solution they may have posted anyway.

How to tell if this is your problem?

Use squidclient to make a web request that bypasses the squid proxy. It should send out the HTTP/1.0 request and get a page back. If the headers of the response include “Transfer-Encoding: chunked” there is your problem.

This is currently only an issue in Squid 2.5 or earlier and 3.0, which is still highly modeled around 2.5.

The solutions are varied depending on your capabilities.

Simplest for some will be to just bypass squid for those domains.

[ UPDATE: (thanks Michael Graham)

Apparently several people are having success with simply dropping the Accept-Encoding header to certain of these broken servers. Adding this to their squid.conf :

# Fix broken sites by removing Accept-Encoding header
acl broken dstdomain …
request_header_access Accept-Encoding deny broken

NP: don’t forget to remove it again when you upgrade out of 3.0

]

Next best is to use peer-routing to divert those domain requests at a squid 2.6 (or if you are feeling experimental a 3.1 build)

If its a serious issue and you are accelerating for one of these broken web servers. Then you will need to stick with Squid 2.6 until 3.1 is available for production use.

Why does it work for 2.6 and 3.1 but not 3.0?

Well, things are a bit messy I’ll have to write it up one day. Suffice to say that 3.1 has a lot more HTTP/1.1 support where the chunked-encoding/decoding was intended for. But 2.6 needed it a bit earlier so a version of the decoding (only!) was done to fit 2.6 needs at solving this same issue for high-performance users earlier last year.

The 3.0 code is just different enough that it would need a whole new back-port project to get it going well. The time and work that would take is being used instead to get 3.1 out faster. Which should be within a month of this writing so procrastinating could solve the problem for you.

[UPDATE: Thanks to the Gentoo Project for their work back-porting this will be available from 3.0.STABLE16-RC1 ]

About these ads

4 Responses to “Chunked Decoding”

  1. Eli Rios » Blog Archive » Página KB da Microsoft retornando em branco no IE7 Says:

    [...] a fonte então. No blog do Squid (http://squidproxy.wordpress.com/2008/04/29/chunked-decoding/) encontrei a causa do problema. Segundo o blog, esse “novo” incidente é causado por [...]

  2. hexodotnet Says:

    I’ve encountered the same problem with 2.6 trying to access msnbc.msn.com. I have Advanced Proxy installed in a Smoothwall 3.0 firewall, and squid is reported as 2.6.STABLE20. (http://www.advproxy.net/changelog.html#swe30)

    The Accept-Encoding fix you noted above worked for me, though.

    http://hexo.net/2008/07/21/problem-loading-msnbc-behind-smoothwall-resolved/

    Thanks!

    -Joe

  3. Henrik Nordström Says:

    The servers having this problem often also forget to include a Vary header in the response. This breaks for example MSIE if configured to use a proxy and not configured to use HTTP/1.1 via proxies. That is a separate issue from the chunked encoding problem. and will be the same even the day Squid is fully HTTP/1.1.

  4. Accept-Encoding http header variations deviations normalize gzip, deflate, identity Says:

    [...] Chunked Decoding [...]

Leave a Reply

Please log in using one of these methods to post your comment:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s


Follow

Get every new post delivered to your Inbox.

Join 32 other followers

%d bloggers like this: