Archive for June, 2007

Beta testing begun.

June 25, 2007

A few days ago I posted the official announcement to the Squid-Dev mailing list and current list of known testers.  It’s time for it to be fully public and told here.

The code has reached a stable enough state that I am now using it full-time myself and have pushed it to Beta testing (pre-pre-PRE release) in the hopes that use by others might find it useful, prove its worthiness, or wrinkle out some more bugs (which, code being code I expect to happen).

Baring a few OS that have known problems:

* Win32 users are out in the cold due to MS choice of stack, even in Vista.

* Debian builds and runs, but does not playing nice to some tests. I’m beginning to think it is the test itself rather than the main squid code.

Squid Deployment Examples

June 20, 2007

I’ve finally put up a couple of example Squid deployments in the real world – Wikipedia and Flickr. They both use Squid as a web accelerator to serve a whole whack of traffic.

I’m looking for some information about current Squid forward proxying setups in the real world – we know that Squid’s capable of a whole lot more than people realise – even given Squid’s shortcomings it can push at least two or three hundred megabits on modern low-end modern hardware without trying too hard – which will save ISPs a whole lot of money even with modern bandwidth prices.

Let me know if you’d like to share your current setup and performance figures. We might even be able to help you make it faster!

Squid-2 /dev/poll update

June 9, 2007

I’ve some feedback from another beta tester of my Solaris /dev/poll support in Squid-2. Quoting his latest email:

“With Squid 2.5-STABLE14, I was seeing 100% CPU on both machines during peak times of the day (mostly pollsys calls), and I couldn’t get more than about 5Mbps of throughput through the infrastructure. This was with approximately 400 users and 10-20 hits/sec.”
“Using 2.HEAD-20070604 with /dev/poll enabled, my internal proxy has dropped to 20% CPU utilization and the external is under 10%, and I can easily get nearly the full throughput of our 45Mbps connection.”

I’ve no idea of the hardware he’s using to achieve this – when I find out I’ll post it here – but thats a definite improvement.

IPv6 sees daylight

June 8, 2007

Still early days in the testing, but squid3-ipv6 now seems to be over its teething segfaults (seems the cache-store was built of some old strange matter) and has been running for a few hours so well.

This post is the first fully browsed to, written and posted from IPv6 🙂

Like so:

::1 -> 2001:5c0:9388:0:217:9aff:febe:30e5(squid) -> (squid) ->
Thank you to everyone who helped get it this far.

The alteration patch is so large, I am planning a daily tarball for the IPv6 code. But that is still a ways off.

For now those continuing with the testing will need to grab clean code from the June 9 daily snapshot and squid3-ipv6 patch when it gets built. Then do a full reconfigure on the binary and destructive rebuild of the testing cache-dir.

This is necessary since most of the segfaults have been traced to a few data store and transfer points causing corruption in transit.

I’m now open for feature-to-be-converted requests.

Squid Sighting: Packeteer “iShared”

June 6, 2007

I came across references to Squid in recent Packeteer iShared release notes (for the FlexInstall, at least.) Inspecting an install of iShared revealed that they’re indeed using Squid for caching in some of their products. Packeteer acquired the iShared technology from Tacit Networks some time ago; but its warming to see Squid used in modern commercial appliances.

Things to look at if websites are hanging!

June 5, 2007

People occasionally ask why site X doesn’t work through their Squid proxy but it does when going direct. Almost every one of those problems can be explained by three phenomena:

  • ECN (Explicit Congestion Notification)
  • PMTUD (Path MTU Discovery)
  • TCP Window Scaling

Explicit Congestion Notification is a method for a router or gateway to signal end-points of a TCP connection that the path is reaching congestion and they should back off a little. ECN uses two bits from the TCP ToS byte to identify if the endpoints are ECN capable and to signal congestion. Unfortunately many older firewalls developed before ECN became main-stream do a variety of silly things when faced with ECN bits – the worst behaviour being simply dropping the packet on the floor. The pre-ECN specification noted those bits were “Not specified” but various firewall vendors took that to mean “must both be zero” and treat non-zero bits there as “invalid traffic.”

Disabling ECN is platform specific and is an operating system parameter – Squid doesn’t have the ability to disable ECN for specific connections. Just use Google to determine how to disable ECN.

Path MTU discovery is a muddy issue. PMTUD is a method for determining when a router/gateway between two IP endpoints has an MTU smaller than the endpoints. For example – if a home PC and server has an MTU of 1500 but the ISP broadband has an MTU of 1480 (as they’re tunneling broadband traffic around their network.) PMTUD specifies that hosts should set the “Don’t Fragment” bit on traffic and routers/gateways should then return an ICMP “Fragmentation Required” message when faced with a packet too large to forward. The sending host receives the ICMP packet, drops its MTU for that IP destination, and resends the packet.

This requires a number of things work:

Firstly, that the router/gateway actually sends the ICMP packet (which doesn’t always happen for various reasons – one of the more popular is a slightly misconfigured network resulting in MTU mismatches at Layer 2; if a packet is dropped by a Layer 2 MTU mismatch no ICMP (which is Layer 3/4) will be generated.)

  • Secondly, that ICMP is not filtered. Unfortunately a number of firewalls and servers have been configured treat ALL ICMP as hostile and drop the traffic. ICMP is a part of the IP infrastructure and is necessary for correct behaviour, but this doesn’t seem to have penetrated the heads of various firewall administrators of various large websites.

Resolving issues relating to PMTUD can be a bit difficult. The correct thing to do is to ensure you allow ICMP to/from your proxy servers, clients and origin servers (which holds true for reverse and forward proxies.) If you have to filter ICMP then filter out the potentially hazardous stuff (echo request/reply, port/host unreachable) but leave the important infrastructure stuff (like “fragmentation required.”) If you’re that stuck, consider dropping the default TCP MSS on your server.

Dropping the TCP MSS is definitely not changing the MTU of the interface. The MTU of your interface(s) should be the same as all other hosts on that network. Most operating systems have the ability to add routes which override the MSS (Maximum Segment Size) for all connections to/from that particular host.

Under Linux you can use the “mss” route flag to do this; eg

“route add default gw X.X.X.X mss 1200”

Note that the MSS is not the same as the IP payload. The MSS is the TCP payload and options. For example, if you have a 1500 byte TCP frame, that will be 20 bytes IP header, 20 bytes TCP header, (say) 20 bytes TCP options and the rest (1440 bytes) for payload. The MSS is the payload (1440 bytes) and the options (20 bytes) = 1460 bytes.

You could just turn off PMTUD and hope that the routers in the path will fragment the packets for you. I don’t do this so I’m not sure how, or how effective this will be.

Finally, TCP Window Scaling is a TCP option which allows TCP to transmit in units larger than the default window size (which is a 16 bit number, so 65535 bytes.) No, this doesn’t mean the packet is 65535 bytes in size; it means it’ll transmit 65535 bytes of data in MSS sized segments before waiting for an ACK.

The TCP WSS option tells the TCP stack how many bits to shift the window size “right” to calculate the true window size. A WSS of 0 shifts the window size to the right by 0 bits, giving the same as normal. A WSS of 1 shifts the window size to the right by 1 bits, giving you a real window size of up to 131072 bytes in 2 byte increments. A WSS of 2 shifts the window size by 2 bits, giving you a window size of 262144 in 4 byte increments, and so on.

This is done so higher TCP throughputs are possible over higher latency links. Higher latency being more than a handful of milliseconds (say 30ms.) Ie, most of the world.

Unfortunately, firewalls strike again. Some firewalls don’t understand the TCP WSS option and will do the most brain damaged thing possible – they’ll simply zero the whole WSS option. This has a horrible side-effect: both sides have sent TCP WSS options, and see the other side sending a TCP WSS option (of 0!), so they assume the option they sent is perfectly fine to use. Unfortunately they send an option of non-zero, and expect the window sizes in TCP frames to be interpreted with an option of zero, and bad things happen. This mostly shows up as slow/stalling TCP connections.

For now, the best thing to do is simply disable TCP WSS. Again, its operating system dependant and you’ll want to use Google to find out how to do it. You can use route flags in some operating systems (such as Linux) to set the maximum TCP window size for destinations if you wish to fine-tune things.

IPv4 shines through into IPv6

June 3, 2007

Squid3-ipv6 performed its full page load across the IPv4-IPv6 boundary two hours ago!

The goal is within sight, blinking orange icons and all.

It’s still operating with limits. There is a bug apparently in the ipcache that causes it to die a few minutes after starting. Also some unchanged socket logic means IPv4 clients are hamstrung and cannot access IPv6 servers yet. Same thing makes split-stack (win32) a long way off.

Still, even one page load is a large milestone for previously dead code. Google never looked so good.