Varnish using malloc, but still writing to disk?

[Jason Price]

Self reply:  Does varnish rewrite the
/var/lib/varnish/<VARNISH_NAME>/_.vsm file periodically?

<googling>

http://www.gossamer-threads.com/lists/varnish/misc/24678

<d'oh!>

Looks like the big control-knob for this is vm.dirty_expire_centisecs
for modern linux kernels.  Every interval, an async copy of the shared
memory log goes to disk.  I'll be evaluating just tweaking this, or
moving it to a tmpfs volume.  yay.

-Jason

On Mon, Apr 27, 2015 at 11:50 AM, Jason Price <japrice at gmail.com> wrote:
> Varnish invocation:
>
> /usr/sbin/varnishd -P /var/run/varnish.pid -a :80 -n vorigin -f
> /etc/varnish/vorigin.vcl -T 127.0.0.1:6082 -t 120 -w 2000,8000,300 -p
> thread_pools=2 -p thread_pool_workspace=100000 -p
> sess_workspace=1000000 -u varnish -g varnish -S /etc/varnish/secret -s
> malloc,90G
>
> We've had a production issue and I've been asserting that varnish
> doesn't write anything to disk, so the root volume IO spike we're
> seeing CAN'T be varnish.
>
> Of course, I look into iotop (specifically 'iotop -obPk') I see
> periodic times where the output is as follows:
>
>   PID  PRIO  USER     DISK READ  DISK WRITE  SWAPIN      IO    COMMAND
>     6 be/4 root        0.00 K/s    0.00 K/s  0.00 % 33.26 % [kworker/u128:0]
>  4185 be/4 varnish     0.00 K/s 6673.66 K/s  0.00 %  0.00 % varnishd
> -P /var/run/varnish.pid -a :80 -n vorigin -f /etc/varnish/vorigin.vcl
> -T 127.0.0.1:6082 -t 120 -w 2000,8000,300 -p thread_pools=2 -p
> thread_pool_workspace=100000 -p sess_workspace=1000000 -u varnish -g
> varnish -S /etc/varnish/secret -s malloc,90G
> Total DISK READ: 0.00 K/s | Total DISK WRITE: 0.00 K/s
>   PID  PRIO  USER     DISK READ  DISK WRITE  SWAPIN      IO    COMMAND
>  4185 be/4 varnish     0.00 K/s 12659.41 K/s  0.00 %  0.00 % varnishd
> -P /var/run/varnish.pid -a :80 -n vorigin -f /etc/varnish/vorigin.vcl
> -T 127.0.0.1:6082 -t 120 -w 2000,8000,300 -p thread_pools=2 -p
> thread_pool_workspace=100000 -p sess_workspace=1000000 -u varnish -g
> varnish -S /etc/varnish/secret -s malloc,90G
>
> I've dug into it with LSOF and sundry, and I can't figure out
> what/where varnish is doing the writing.  It's not constant... 10-20
> seconds will pass before I see it again for several seconds (2-8ish).
>
> NOTE: normal load for these boxes is 3-10k req/sec.
>
> Any thoughts?  Why is varnish writing to disk (more to the point:
> writing tens megabytes at a time)?
>
> varnish version 3.0.5
> Amazon Linux
> Kernel 3.14.35-28
>
> Thank you for any help;
> -Jason