varnishncsa logs split per domain

[Andrei]

Hello,

By not using splitlogs or an intermediary script, you will be forced to run
multiple instances of varnishncsa, which isn't optimal if you host multiple
domains. The more traffic/domains you have, the more resources you will
consume on parsing the same data across multiple channels. Yes, varnishncsa
supports VSL, however I think your approach is a bit off (no offense). As
in, you need to shift the log writing process away from varnishncsa. By
doing so, you only have one instance of varnishncsa using resources to
gather the data, which is then fed to a parser that handles the per domain
log splits and writes. That's where 'splitlogs' came into play.

As your question does raise some interest in the cPanel community (myself
and Miguel González on this list for example), I threw together a quick
Perl script that will in short, pipe and parse data between varnishncsa and
the splitlogs binary for cache hits. This lets splitlogs handle the queued
log writes which are later parsed for cPanel bandwidth usage and graphs,
webalizer, awstats, logaholic, etc - https://github.com/AndreiG6/vscp



On Sun, Nov 13, 2016 at 12:18 PM, Admin Beckspaced <admin at beckspaced.com>
wrote:

> Same Hello here ;)
>
> did have a more in depth look in the manual and figured out that
> varnishncsa does support VSL query.
> so someone could filter on the Request Header and Host
>
> varnishncsa -q "ReqHeader ~ '^Host: .*example.com'"
>
> which would produce a log for a specific domain only
>
> it then would need multiple varnishncsa instances for logging per domain,
> which I found here:
>
> https://kevops.com/2015/11/varnish-logging-per-host-with-init-script/
>
> I use varnish version 5 and then there would be no need for splitlog and
> the logs would be created directly.
>
> please correct me if I'm wrong?
>
> thanks for your time & help
> Becki
>
>
> Am 12.11.2016 um 17:05 schrieb Andrei:
>
>> Hello again,
>>
>> My apologies for not explaining my thoughts better earlier then. Afaik,
>> varnishncsa does not have a native method to split output based on
>> different parameters. The method I was thinking of was based on piping
>> varnishncsa output through splitlogs (or similar) for the log processing
>> and writeouts. Since replying earlier, I've got this working on a cPanel
>> server with piped logging enabled for Apache using the following two for
>> example (X-Port is a custom header set in vcl_recv related to SSL
>> offloading, but you can use a static value or similar custom header):
>>
>> varnishncsa -F "%{HOST}i:%{X-Port}i %h %l %u %t \"%m %U%q %H\" %s %b
>> \"%{Referer}i\" \"%{User-agent}i\""|sed -e
>> 's#^www\.##g'|/usr/local/cpanel/bin/splitlogs --main=`hostname`
>> --mainout=/usr/local/apache/logs/access_log
>> varnishncsa -F "%{HOST}i %{%s}t %b ."|sed -e
>> 's#^www\.##g'|/usr/local/cpanel/bin/splitlogs --main=`hostname`
>> --suffix=-bytes_log
>>
>> The above pipes the requests to the splitlogs binary which queues then
>> writes to separate logs per domain, that are later processed by the
>> cPanel log stats apps. Either way, I believe you need an intermediary
>> script to queue and write the log entries per domain. While looking into
>> this process, I ran across this little tidbit which you may find of use
>> https://gist.github.com/garlandkr/4954272 for logstash style output.
>>
>>
>>
>
> _______________________________________________
> varnish-misc mailing list
> varnish-misc at varnish-cache.org
> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/varnish-misc/attachments/20161113/5d289a37/attachment.html>