Logfile improvements in Squid-2-HEAD

by

I’ve committed my logfile handling improvements to Squid-2-HEAD. Essentially, it lets people write self-contained code modules to implement different logging methods. The three supported methods now are:

  • STDIO, which is how Squid currently does its logging;
  • Syslog, which is compiled in if you enable it; and
  • Daemon, which uses a simple external helper to write logfiles to disk.

Those of you who have run Squid may have noticed that it couldn’t support writing more than a hundred or so requests a second to disk before performance suffered. There’s no reason it shouldn’t handle this – a hundred requests a second is only 16 kilobytes a second to write – but the use of STDIO routines to do this had a negative impact on performance.

The logfile daemon allows the blocking disk IO to occur outside of the main Squid process; which basically means Squid can continue doing what its doing well (all the other stuff) and any blocking disk activity occurs in a seperate process.

To use? Compile and install Squid-2-HEAD, then include the following line into your configuration:

logtype daemon

In reality, Squid with the logging daemon can now handle writing -thousands of requests a second- to disk without any performance impact. Furthermore, if the logging daemon can’t write to disk fast enough Squid will log a error message stating its falling behind and drop logging entries.

I’ve tested this up to three thousand requests a second over the course of a few hours (to a dedicated logging disk however) and it handles it without a problem.

If enterprising souls wished, they could write a UDP logging helper, or a MySQL external logging helper, without needing to modify the Squid codebase.

This code will eventually also appear in Squid-3 after 3.0 is released.

Advertisements

Leave a Reply

Please log in using one of these methods to post your comment:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s


%d bloggers like this: