varnishncsa logs split per domain

[Guillaume Quintard]

Is the resources argument that compelling? I find it cleaner to have one
ncsa per domain, plus the ncsas will read from memory, which is super fast.

Sure you can then split logs afterwards using awk or whatever, but that's
adding an extra layer and serializing a process that doesn't need to be.

But I'm not an admin, so I may be off.

On Nov 13, 2016 16:27, "Andrei" <lagged at gmail.com> wrote:

> Hello,
>
> By not using splitlogs or an intermediary script, you will be forced to
> run multiple instances of varnishncsa, which isn't optimal if you host
> multiple domains. The more traffic/domains you have, the more resources you
> will consume on parsing the same data across multiple channels.
> Yes, varnishncsa supports VSL, however I think your approach is a bit off
> (no offense). As in, you need to shift the log writing process away
> from varnishncsa. By doing so, you only have one instance of varnishncsa
> using resources to gather the data, which is then fed to a parser that
> handles the per domain log splits and writes. That's where 'splitlogs' came
> into play.
>
> As your question does raise some interest in the cPanel community (myself
> and Miguel González on this list for example), I threw together a quick
> Perl script that will in short, pipe and parse data between varnishncsa and
> the splitlogs binary for cache hits. This lets splitlogs handle the queued
> log writes which are later parsed for cPanel bandwidth usage and graphs,
> webalizer, awstats, logaholic, etc - https://github.com/AndreiG6/vscp
>
>
>
> On Sun, Nov 13, 2016 at 12:18 PM, Admin Beckspaced <admin at beckspaced.com>
> wrote:
>
>> Same Hello here ;)
>>
>> did have a more in depth look in the manual and figured out that
>> varnishncsa does support VSL query.
>> so someone could filter on the Request Header and Host
>>
>> varnishncsa -q "ReqHeader ~ '^Host: .*example.com'"
>>
>> which would produce a log for a specific domain only
>>
>> it then would need multiple varnishncsa instances for logging per domain,
>> which I found here:
>>
>> https://kevops.com/2015/11/varnish-logging-per-host-with-init-script/
>>
>> I use varnish version 5 and then there would be no need for splitlog and
>> the logs would be created directly.
>>
>> please correct me if I'm wrong?
>>
>> thanks for your time & help
>> Becki
>>
>>
>> Am 12.11.2016 um 17:05 schrieb Andrei:
>>
>>> Hello again,
>>>
>>> My apologies for not explaining my thoughts better earlier then. Afaik,
>>> varnishncsa does not have a native method to split output based on
>>> different parameters. The method I was thinking of was based on piping
>>> varnishncsa output through splitlogs (or similar) for the log processing
>>> and writeouts. Since replying earlier, I've got this working on a cPanel
>>> server with piped logging enabled for Apache using the following two for
>>> example (X-Port is a custom header set in vcl_recv related to SSL
>>> offloading, but you can use a static value or similar custom header):
>>>
>>> varnishncsa -F "%{HOST}i:%{X-Port}i %h %l %u %t \"%m %U%q %H\" %s %b
>>> \"%{Referer}i\" \"%{User-agent}i\""|sed -e
>>> 's#^www\.##g'|/usr/local/cpanel/bin/splitlogs --main=`hostname`
>>> --mainout=/usr/local/apache/logs/access_log
>>> varnishncsa -F "%{HOST}i %{%s}t %b ."|sed -e
>>> 's#^www\.##g'|/usr/local/cpanel/bin/splitlogs --main=`hostname`
>>> --suffix=-bytes_log
>>>
>>> The above pipes the requests to the splitlogs binary which queues then
>>> writes to separate logs per domain, that are later processed by the
>>> cPanel log stats apps. Either way, I believe you need an intermediary
>>> script to queue and write the log entries per domain. While looking into
>>> this process, I ran across this little tidbit which you may find of use
>>> https://gist.github.com/garlandkr/4954272 for logstash style output.
>>>
>>>
>>>
>>
>> _______________________________________________
>> varnish-misc mailing list
>> varnish-misc at varnish-cache.org
>> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
>>
>
>
> _______________________________________________
> varnish-misc mailing list
> varnish-misc at varnish-cache.org
> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/varnish-misc/attachments/20161113/f952515c/attachment.html>