Squid-2 updates: Logfiles and buffers

by

I’ve made three changes to the Squid-2.HEAD codebase this weekend.

First up – I’ve modified the memory allocator to not zero every sort of buffer. This can be quite expensive for large buffers, especially on older machines or very busy Squid servers. Squid-2.HEAD now has the “zero_buffers” option which currently defaults to “on”. To disable zero’ing buffers please add “zero_buffers off” to your squid.conf file. I’ve seen up to 10% CPU savings on my testbed at home but this may vary wildly depending upon work load.

Secondly – the ‘logtype’ configuration option has been removed and replaced with the ability to define logging types per logfile. You can now prefix your log line with “daemon:”, “stdio:”, “udp:” or “syslog:”. “syslog:” works the same as before; “stdio:” and “daemon:” just take a path, and “udp:” takes an IP:port URL.

To log to a UDP socket, try:

access_log udp://192.168.1.101:1234

Please note though that the default UDP payload size (defined in src/logging_mod_udp.c) is 1400 bytes and any application you decide to use to dump the logfile entries must be able to receive UDP packets that big. There’s a system-wide UDP packet limit in some operating systems (for example, sysctl net.inet.udp.maxdgram under FreeBSD) to also consider. If in doubt, do a tcpdump on both sides and make sure you’re seeing the packets of the right size getting there.

Note too you can’t use these options for the cache_log – it must always be a normal file path.

Advertisements

One Response to “Squid-2 updates: Logfiles and buffers”

  1. mmarch Says:

    Hi Adrian,
    I’ve written a simple perl script which logs on a mysql table. I’ve put it together quickly to see if/how the whole thing worked, and I’m planning to refine it on the next days.

    Relevant squid configuration directives:

    access_log daemon:/var/log/squid/access.log squid
    logfile_daemon /usr/squid/libexec/logfile-daemon_mysql.pl

    MySQL script:

    CREATE DATABASE squid_devel_log;
    GRANT INSERT,SELECT ON squid_devel_log.* TO ‘squid’@’localhost’ IDENTIFIED BY ‘squid’;


    — This table is based on squid’s default ‘squid’ logformat, with minor modifications
    — (the two slashes are removed)
    — original:
    — logformat squid %ts.%03tu %6tr %>a %Ss/%03Hs %<st %rm %ru %un %Sh/%a %Ss %03Hs %<st %rm %ru %un %Sh %connect($dsn, $user, $pass, { AutoCommit => 1, RaiseError => 1, PrintError => 1 }) or die $DBI::errstr;

    my $sth = $dbh->prepare(“INSERT INTO $table VALUES(NULL,?,?,?,?,?,?,?,?,?,?,?,?)”);

    while (my $line = ) {
    chomp $line;

    my $cmd = substr($line, 0, 1); # command
    substr($line, 0, 1, ‘ ‘); # substitute the command byte with a blank
    if ( $cmd eq ‘L’ ) {
    my @values = split / \s+ /xms, $line;
    shift @values; # the first blank generates an empty bind value that has to be removed
    eval {
    $sth->execute(@values) or die $sth->errstr
    };
    if ( $@ ) {
    warn $@ . ” values=(” . join(‘,’, @values) . “)”;
    }
    }
    }

    $dbh->disconnect();

    As you can see, it’s very primitive, but I’ve tested on two systems of mine and it works.
    Some things that I’ll have to address shortly:
    – sql data types: now they’re all strings, but some of them are better stored directly as numbers, to allow correct sorting, calculations, etc. (e.g. avg(response_time), max(response_time))
    – currently the path is ignored: it could be used to specify the database connection details directly as a dsn or via some sort of custom syntax (e.g. to allow passing username and password)
    – or the path could be used to point to a configuration file which would hold in a simple format (key: value, e.g. yaml) the connection details and maybe some other directive for the perl script.
    – on startup, test the database via some simple SELECT statements and warn() the user if it’s not properly setup, so e.g. one doesn’t get a squid crash if the table structure is wrong because the log script die()s.
    – implement the other command-codes. One idea is for example to implement the “rotate logs” command as something like “calculate summary data based on the current log records, store that data in a “summary” table) and delete all the log records”

    Thanks for your work on squid!

    Marcello

Leave a Reply

Please log in using one of these methods to post your comment:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s


%d bloggers like this: