Squid-2 updates: Logfiles and buffers
I’ve made three changes to the Squid-2.HEAD codebase this weekend.
First up - I’ve modified the memory allocator to not zero every sort of buffer. This can be quite expensive for large buffers, especially on older machines or very busy Squid servers. Squid-2.HEAD now has the “zero_buffers” option which currently defaults to “on”. To disable zero’ing buffers please add “zero_buffers off” to your squid.conf file. I’ve seen up to 10% CPU savings on my testbed at home but this may vary wildly depending upon work load.
Secondly - the ‘logtype’ configuration option has been removed and replaced with the ability to define logging types per logfile. You can now prefix your log line with “daemon:”, “stdio:”, “udp:” or “syslog:”. “syslog:” works the same as before; “stdio:” and “daemon:” just take a path, and “udp:” takes an IP:port URL.
To log to a UDP socket, try:
access_log udp://192.168.1.101:1234
Please note though that the default UDP payload size (defined in src/logging_mod_udp.c) is 1400 bytes and any application you decide to use to dump the logfile entries must be able to receive UDP packets that big. There’s a system-wide UDP packet limit in some operating systems (for example, sysctl net.inet.udp.maxdgram under FreeBSD) to also consider. If in doubt, do a tcpdump on both sides and make sure you’re seeing the packets of the right size getting there.
Note too you can’t use these options for the cache_log - it must always be a normal file path.
February 20, 2008 at 6:54 pm
Hi Adrian,
I’ve written a simple perl script which logs on a mysql table. I’ve put it together quickly to see if/how the whole thing worked, and I’m planning to refine it on the next days.
Relevant squid configuration directives:
access_log daemon:/var/log/squid/access.log squid
logfile_daemon /usr/squid/libexec/logfile-daemon_mysql.pl
MySQL script:
CREATE DATABASE squid_devel_log;
GRANT INSERT,SELECT ON squid_devel_log.* TO ’squid’@'localhost’ IDENTIFIED BY ’squid’;
–
– This table is based on squid’s default ’squid’ logformat, with minor modifications
– (the two slashes are removed)
– original:
– logformat squid %ts.%03tu %6tr %>a %Ss/%03Hs %<st %rm %ru %un %Sh/%a %Ss %03Hs %<st %rm %ru %un %Sh %connect($dsn, $user, $pass, { AutoCommit => 1, RaiseError => 1, PrintError => 1 }) or die $DBI::errstr;
my $sth = $dbh->prepare(”INSERT INTO $table VALUES(NULL,?,?,?,?,?,?,?,?,?,?,?,?)”);
while (my $line = ) {
chomp $line;
my $cmd = substr($line, 0, 1); # command
substr($line, 0, 1, ‘ ‘); # substitute the command byte with a blank
if ( $cmd eq ‘L’ ) {
my @values = split / \s+ /xms, $line;
shift @values; # the first blank generates an empty bind value that has to be removed
eval {
$sth->execute(@values) or die $sth->errstr
};
if ( $@ ) {
warn $@ . ” values=(” . join(’,', @values) . “)”;
}
}
}
$dbh->disconnect();
As you can see, it’s very primitive, but I’ve tested on two systems of mine and it works.
Some things that I’ll have to address shortly:
- sql data types: now they’re all strings, but some of them are better stored directly as numbers, to allow correct sorting, calculations, etc. (e.g. avg(response_time), max(response_time))
- currently the path is ignored: it could be used to specify the database connection details directly as a dsn or via some sort of custom syntax (e.g. to allow passing username and password)
- or the path could be used to point to a configuration file which would hold in a simple format (key: value, e.g. yaml) the connection details and maybe some other directive for the perl script.
- on startup, test the database via some simple SELECT statements and warn() the user if it’s not properly setup, so e.g. one doesn’t get a squid crash if the table structure is wrong because the log script die()s.
- implement the other command-codes. One idea is for example to implement the “rotate logs” command as something like “calculate summary data based on the current log records, store that data in a “summary” table) and delete all the log records”
Thanks for your work on squid!
Marcello