Comments for The Squid Web Proxy/Cache Blog

Comment on Things to look at if websites are hanging! by Chunked Decoding « The Squid Web Proxy/Cache Blog

Chunked Decoding « The Squid Web Proxy/Cache Blog — Tue, 29 Apr 2008 14:24:11 +0000

[...] familiar to some yes? I’m bringing it up now because while it is an old problem, its not the TCP issues Adrian wrote about earlier and you should also check if you find its not this. Which incidentally can have exactly the same [...]

Comment on The Squid Configuration Manual by revconnect

revconnect — Thu, 27 Mar 2008 01:49:20 +0000

Thanks For The Config Manual

Comment on Squid-2 updates: Logfiles and buffers by mmarch

mmarch — Wed, 20 Feb 2008 18:54:54 +0000

Hi Adrian,
I’ve written a simple perl script which logs on a mysql table. I’ve put it together quickly to see if/how the whole thing worked, and I’m planning to refine it on the next days.

Relevant squid configuration directives:

access_log daemon:/var/log/squid/access.log squid
logfile_daemon /usr/squid/libexec/logfile-daemon_mysql.pl

MySQL script:

CREATE DATABASE squid_devel_log;
GRANT INSERT,SELECT ON squid_devel_log.* TO ’squid’@'localhost’ IDENTIFIED BY ’squid’;

–
– This table is based on squid’s default ’squid’ logformat, with minor modifications
– (the two slashes are removed)
– original:
– logformat squid %ts.%03tu %6tr %>a %Ss/%03Hs %a %Ss %03Hs %connect($dsn, $user, $pass, { AutoCommit => 1, RaiseError => 1, PrintError => 1 }) or die $DBI::errstr;

my $sth = $dbh->prepare(”INSERT INTO $table VALUES(NULL,?,?,?,?,?,?,?,?,?,?,?,?)”);

while (my $line = ) {
chomp $line;

my $cmd = substr($line, 0, 1); # command
substr($line, 0, 1, ‘ ‘); # substitute the command byte with a blank
if ( $cmd eq ‘L’ ) {
my @values = split / \s+ /xms, $line;
shift @values; # the first blank generates an empty bind value that has to be removed
eval {
$sth->execute(@values) or die $sth->errstr
};
if ( $@ ) {
warn $@ . ” values=(” . join(’,', @values) . “)”;
}
}
}

$dbh->disconnect();

As you can see, it’s very primitive, but I’ve tested on two systems of mine and it works.
Some things that I’ll have to address shortly:
- sql data types: now they’re all strings, but some of them are better stored directly as numbers, to allow correct sorting, calculations, etc. (e.g. avg(response_time), max(response_time))
- currently the path is ignored: it could be used to specify the database connection details directly as a dsn or via some sort of custom syntax (e.g. to allow passing username and password)
- or the path could be used to point to a configuration file which would hold in a simple format (key: value, e.g. yaml) the connection details and maybe some other directive for the perl script.
- on startup, test the database via some simple SELECT statements and warn() the user if it’s not properly setup, so e.g. one doesn’t get a squid crash if the table structure is wrong because the log script die()s.
- implement the other command-codes. One idea is for example to implement the “rotate logs” command as something like “calculate summary data based on the current log records, store that data in a “summary” table) and delete all the log records”

Thanks for your work on squid!

Marcello

Comment on How cachable is google (part 2) - Youtube content by chudycebu

chudycebu — Fri, 08 Feb 2008 16:40:37 +0000

I’ve been trying to look for this squid that has this storeurl_rewrite features and found out that is still on development. Currently using Windows XP squid-2.6.STABLE18 (10 Jan 200

I just want to cache the youtube videos (bec its #1 bandwidth sucker and #2 is the imeem) refresh pattern get_video just works fine for 3years until last thursday. Youtube has been upgraded their url video files.

They add signature, your ip, ipbits, expire, and key in url video files. so it no longer be cache bec signature always change. so i try to use urlrewrite temporarity until storeurlrewrite feature is up.

$| = 1;

while () {
chomp;
# print STDERR $_ . “\n”;
if (m/^http:\/\/([A-Za-z]*?)-(.*?)\.(.*)\.youtube\.com\/get_video\?video_id=(.*)\&signature=(.*)\&ip=(.*)\&ipbits=(.*)\&expire=(.*)\&key=(.*) /) {
print “http://” . $1 . “-” . $2 . “.” . $3 . “.youtube.com/get_video?video_id=” . $4 . “\n”;
} elsif (m/^http:\/\/(.*?)\/get_video\?video_id=(.*)\&origin=(.*)\&signature=(.*)\&ip=(.*)\&ipbits=(.*)\&expire=(.*)\&key=(.*) /) {
print “http://” . $3 . “/get_video?video_id=” . $2 . “\n”;
} else {
print $_ . “\n”;
}
}

i just bypass the CDN network and use the origin url from youtube.
if the storeurlrewrite feature i really love to cache imeem(which is the #2 most annoying bandwidth sucker)

Comment on Whats going on with Squid-2 and Squid-3 ? by Adrian Chadd

Adrian Chadd — Thu, 10 Jan 2008 22:38:58 +0000

tgbyhn,

Varnish is a fun project.

phk has plenty of great ideas, but the Varnish design is tailored towards a specific workload (ie mostly fits in RAM) which handles small objects. Varnish doesn’t take advantage of things that Web Caches should (like object locality on disk) and, well, Squid doesn’t either, but Varnish actively avoids doing so by putting all its faith in the operating system VM system. You can’t scale specific workloads that way.

There’s nothing wrong with the Squid design per se. Plenty of high performance applications look like squid do, they just don’t copy data around as much.

Squid, in its core, is just a big event processing loop. Eventually this’ll have to be modified to support SMP but the basic theory is fine. What we need to do is change how we work with the disk and network to be more efficient and thats precisely what I’m doing.

Ideally Squid in ~ 12 months will look at lot more interesting. Just realise that phk and des are funded to do this and I’m mostly doing this for free.

Comment on Whats going on with Squid-2 and Squid-3 ? by Adrian Chadd

Adrian Chadd — Thu, 10 Jan 2008 22:35:05 +0000

ohernandez;

Squid doesn’t currently support RTSP. We’d love to support it but it’ll take time to code up and none of us actually work for ISPs or companies that have these requirements. We basically need to find funding to code this stuff up.

Send me an email if you’re interested in looking into helping with this.

Comment on Whats going on with Squid-2 and Squid-3 ? by ohernandez

ohernandez — Thu, 10 Jan 2008 20:08:55 +0000

Hello,
Squid currently support RTSP ?
i can’t access to view windows media, you know alternative for doing this ?

Comment on Whats going on with Squid-2 and Squid-3 ? by tgbyhn

tgbyhn — Thu, 10 Jan 2008 17:58:11 +0000

I like the idea of reducing copying between buffers. What do you think of the Varnish design. The Varnish design takes copy avoidance to the extreme.

The Varnish developer claims that the Squid design is something that should be avoided. I would be really interested in hearing your take.

Comment on How cachable is google (part 2) - Youtube content by Adrian Chadd

Adrian Chadd — Tue, 25 Dec 2007 15:03:59 +0000

The legality isn’t my concern really. This stuff isn’t prefetching Google Earth, nor is it distributing “map packs” to make things faster. Its simply caching the content like any other content. Google isn’t specifically trying to make the map tile images uncachable.

Comment on How cachable is google (part 2) - Youtube content by phreaki

phreaki — Thu, 20 Dec 2007 03:25:24 +0000

Thanks for working on this very important task!

Small and third pipe operators need this type of caching to offset P2P, so I’ll be trying this method out soon.

I ponder however: Google Earth? I don’t know if it uses SSL, but others have gotten into trouble for making ‘map packs’. I hope the DMCA could protect like it’s intended for those that try to make everything uncacheable.