Varnish using malloc, but still writing to disk?

Jason Price japrice at gmail.com
Mon Apr 27 17:50:23 CEST 2015


Varnish invocation:

/usr/sbin/varnishd -P /var/run/varnish.pid -a :80 -n vorigin -f
/etc/varnish/vorigin.vcl -T 127.0.0.1:6082 -t 120 -w 2000,8000,300 -p
thread_pools=2 -p thread_pool_workspace=100000 -p
sess_workspace=1000000 -u varnish -g varnish -S /etc/varnish/secret -s
malloc,90G

We've had a production issue and I've been asserting that varnish
doesn't write anything to disk, so the root volume IO spike we're
seeing CAN'T be varnish.

Of course, I look into iotop (specifically 'iotop -obPk') I see
periodic times where the output is as follows:

  PID  PRIO  USER     DISK READ  DISK WRITE  SWAPIN      IO    COMMAND
    6 be/4 root        0.00 K/s    0.00 K/s  0.00 % 33.26 % [kworker/u128:0]
 4185 be/4 varnish     0.00 K/s 6673.66 K/s  0.00 %  0.00 % varnishd
-P /var/run/varnish.pid -a :80 -n vorigin -f /etc/varnish/vorigin.vcl
-T 127.0.0.1:6082 -t 120 -w 2000,8000,300 -p thread_pools=2 -p
thread_pool_workspace=100000 -p sess_workspace=1000000 -u varnish -g
varnish -S /etc/varnish/secret -s malloc,90G
Total DISK READ: 0.00 K/s | Total DISK WRITE: 0.00 K/s
  PID  PRIO  USER     DISK READ  DISK WRITE  SWAPIN      IO    COMMAND
 4185 be/4 varnish     0.00 K/s 12659.41 K/s  0.00 %  0.00 % varnishd
-P /var/run/varnish.pid -a :80 -n vorigin -f /etc/varnish/vorigin.vcl
-T 127.0.0.1:6082 -t 120 -w 2000,8000,300 -p thread_pools=2 -p
thread_pool_workspace=100000 -p sess_workspace=1000000 -u varnish -g
varnish -S /etc/varnish/secret -s malloc,90G

I've dug into it with LSOF and sundry, and I can't figure out
what/where varnish is doing the writing.  It's not constant... 10-20
seconds will pass before I see it again for several seconds (2-8ish).

NOTE: normal load for these boxes is 3-10k req/sec.

Any thoughts?  Why is varnish writing to disk (more to the point:
writing tens megabytes at a time)?

varnish version 3.0.5
Amazon Linux
Kernel 3.14.35-28

Thank you for any help;
-Jason



More information about the varnish-misc mailing list