Varnishncsa Random Log Sampling ?

[Dridi Boukelmoune]

On Mon, Mar 2, 2020 at 11:15 AM Yassine Aouadi
<yassine.aouadi90 at gmail.com> wrote:
>
>
> Hello,
>
> I  am sending my Varnish log to  to remote SAAS solution  I and want to improve  logs costs by  implementing a server side sampling solution .
>
> First I splitted Varnishnncsa  into two service one for error logs and the other for acces logs  :
>
> CGroup: /system.slice/varnishncsa-error.service
>            └─18458 /usr/bin/varnishncsa -c -b -a -w /var/log/varnish/varnishncsa-error.log -D -P /run/varnishncsa/varnishncsa-error.pid -f /etc/varnish/varnishncsa_logmatic.format -q *Status > 399
>
>   CGroup: /system.slice/varnishncsa.service
>            └─18347 /usr/bin/varnishncsa -c -b -a -w /var/log/varnish/varnishncsa-access.log -D -P /run/varnishncsa/varnishncsa-access.pid -f /etc/varnish/varnishncsa_logmatic.format -q *Status < 400
>
>
> Is there Any way to go further with  varnishncsa  and perform and random sampling of my access logs ? for example write only 10 % of access logs
>
> If it's not possible with varnishncsa any   Suggestion ? I tried rsyslog random sampling but  I am facing memory leaks while stress testing server with high load

Hi,

I think the closest to what you want is rate limiting, see the
documentation for the varnishstat -R option. Otherwise you can always
do the sampling one step downstream and instead of sending
varnishncsa-<something>.log whenever logrotate triggers a rotation you
run script that sends 1 line every 10 lines. But I think rate limiting
with -R is simpler and instead of a percentage that depends highly on
your traffic you can actually get a limit according to a budget since
you wish to reduce costs.

Dridi