varnishncsa logs split per domain

[Andrei]

I haven't done any actual benchmarks for the resource usage, however as an
admin I would rather not have numerous processes for a single data channel
that needs to be parsed and split.

My apologies for sort of hijacking this thread with a cPanel integration.
It's just so widely used, that I'm sure others here have had the need or
thought at some point and figured I would share my findings. As cPanel
servers are known to house hundreds of domains, per domain varnishncsa
simply isn't an option, which is why I chose to piggyback on splitlogs as
it's already a supported option of theirs.

On Nov 13, 2016 17:44, "Guillaume Quintard" <guillaume at varnish-software.com>
wrote:

> Is the resources argument that compelling? I find it cleaner to have one
> ncsa per domain, plus the ncsas will read from memory, which is super fast.
>
> Sure you can then split logs afterwards using awk or whatever, but that's
> adding an extra layer and serializing a process that doesn't need to be.
>
> But I'm not an admin, so I may be off.
>
> On Nov 13, 2016 16:27, "Andrei" <lagged at gmail.com> wrote:
>
>> Hello,
>>
>> By not using splitlogs or an intermediary script, you will be forced to
>> run multiple instances of varnishncsa, which isn't optimal if you host
>> multiple domains. The more traffic/domains you have, the more resources you
>> will consume on parsing the same data across multiple channels.
>> Yes, varnishncsa supports VSL, however I think your approach is a bit off
>> (no offense). As in, you need to shift the log writing process away
>> from varnishncsa. By doing so, you only have one instance of varnishncsa
>> using resources to gather the data, which is then fed to a parser that
>> handles the per domain log splits and writes. That's where 'splitlogs' came
>> into play.
>>
>> As your question does raise some interest in the cPanel community (myself
>> and Miguel González on this list for example), I threw together a quick
>> Perl script that will in short, pipe and parse data between varnishncsa and
>> the splitlogs binary for cache hits. This lets splitlogs handle the queued
>> log writes which are later parsed for cPanel bandwidth usage and graphs,
>> webalizer, awstats, logaholic, etc - https://github.com/AndreiG6/vscp
>>
>>
>>
>> On Sun, Nov 13, 2016 at 12:18 PM, Admin Beckspaced <admin at beckspaced.com>
>> wrote:
>>
>>> Same Hello here ;)
>>>
>>> did have a more in depth look in the manual and figured out that
>>> varnishncsa does support VSL query.
>>> so someone could filter on the Request Header and Host
>>>
>>> varnishncsa -q "ReqHeader ~ '^Host: .*example.com'"
>>>
>>> which would produce a log for a specific domain only
>>>
>>> it then would need multiple varnishncsa instances for logging per
>>> domain, which I found here:
>>>
>>> https://kevops.com/2015/11/varnish-logging-per-host-with-init-script/
>>>
>>> I use varnish version 5 and then there would be no need for splitlog and
>>> the logs would be created directly.
>>>
>>> please correct me if I'm wrong?
>>>
>>> thanks for your time & help
>>> Becki
>>>
>>>
>>> Am 12.11.2016 um 17:05 schrieb Andrei:
>>>
>>>> Hello again,
>>>>
>>>> My apologies for not explaining my thoughts better earlier then. Afaik,
>>>> varnishncsa does not have a native method to split output based on
>>>> different parameters. The method I was thinking of was based on piping
>>>> varnishncsa output through splitlogs (or similar) for the log processing
>>>> and writeouts. Since replying earlier, I've got this working on a cPanel
>>>> server with piped logging enabled for Apache using the following two for
>>>> example (X-Port is a custom header set in vcl_recv related to SSL
>>>> offloading, but you can use a static value or similar custom header):
>>>>
>>>> varnishncsa -F "%{HOST}i:%{X-Port}i %h %l %u %t \"%m %U%q %H\" %s %b
>>>> \"%{Referer}i\" \"%{User-agent}i\""|sed -e
>>>> 's#^www\.##g'|/usr/local/cpanel/bin/splitlogs --main=`hostname`
>>>> --mainout=/usr/local/apache/logs/access_log
>>>> varnishncsa -F "%{HOST}i %{%s}t %b ."|sed -e
>>>> 's#^www\.##g'|/usr/local/cpanel/bin/splitlogs --main=`hostname`
>>>> --suffix=-bytes_log
>>>>
>>>> The above pipes the requests to the splitlogs binary which queues then
>>>> writes to separate logs per domain, that are later processed by the
>>>> cPanel log stats apps. Either way, I believe you need an intermediary
>>>> script to queue and write the log entries per domain. While looking into
>>>> this process, I ran across this little tidbit which you may find of use
>>>> https://gist.github.com/garlandkr/4954272 for logstash style output.
>>>>
>>>>
>>>>
>>>
>>> _______________________________________________
>>> varnish-misc mailing list
>>> varnish-misc at varnish-cache.org
>>> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
>>>
>>
>>
>> _______________________________________________
>> varnish-misc mailing list
>> varnish-misc at varnish-cache.org
>> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/varnish-misc/attachments/20161113/2dcb6aa0/attachment-0001.html>