Multiple log files for one instance

[Michael Alger]

On Tue, Sep 14, 2010 at 05:53:04PM -0500, Justin Pasher wrote:
>
> I want to setup a generic Varnish instance that will be used to
> handle multiple web sites. All of the web sites are directed to
> the same backend and the correct site is served up by Apache
> using Virtual Hosts that match the corresponding Host: header.
> The idea is that a new site can have caching "turned on" simply
> by changing its IP address from the live server to the Varnish
> server. However, by doing so, all requests come into the same
> instance. This means that I cannot create separate log files for
> each site using varnishncsa.
>
> Can anyone think of a way to do something like this? Thanks.

I do this by writing a single log, and then splitting it every hour
into the relevant individual logs. This might not be the most
optimum way of doing it, but it was the easiest way to convert from
our squid setup (where squid itself logged to multiple logfiles)
without interfering with all my other scripts.

The basic idea is to use

  varnishlog -c -a -w /var/log/client.log -D -P /var/run/varnishlog.pid

and then periodically invoke another script which moves this
"client.log" to a temporary directory and sends SIGHUP to the
varnishlog process so it recreates a new logfile.

Then, a Perl script uses

  varnishncsa -r /var/log/temp/client.log

to read through the logfile, matching the hostname in each line and
writing that to an appropriate file.

After that's done, they're compressed and moved into another
directory where they can be picked up by our log analysis and
storage server.

If you'd like to take this approach I can provide you with the
scripts I'm using; written in Bash for the basic file handling and
Perl for the log splitter. The limitation which will potentially hit
you is my splitter script has a configuration section which defines
which hostnames map to which logfile, with everything not explicitly
matched going to a generic logfile.

You could potentially run a varnishncsa to read the live log data,
and split it on-the-fly. My system was built around an hourly
collection of logfiles, so splitting it periodically hasn't made any
difference to the functionality we get.