Varnish using malloc, but still writing to disk?
Jason Price
japrice at gmail.com
Mon Apr 27 17:50:23 CEST 2015
Varnish invocation:
/usr/sbin/varnishd -P /var/run/varnish.pid -a :80 -n vorigin -f
/etc/varnish/vorigin.vcl -T 127.0.0.1:6082 -t 120 -w 2000,8000,300 -p
thread_pools=2 -p thread_pool_workspace=100000 -p
sess_workspace=1000000 -u varnish -g varnish -S /etc/varnish/secret -s
malloc,90G
We've had a production issue and I've been asserting that varnish
doesn't write anything to disk, so the root volume IO spike we're
seeing CAN'T be varnish.
Of course, I look into iotop (specifically 'iotop -obPk') I see
periodic times where the output is as follows:
PID PRIO USER DISK READ DISK WRITE SWAPIN IO COMMAND
6 be/4 root 0.00 K/s 0.00 K/s 0.00 % 33.26 % [kworker/u128:0]
4185 be/4 varnish 0.00 K/s 6673.66 K/s 0.00 % 0.00 % varnishd
-P /var/run/varnish.pid -a :80 -n vorigin -f /etc/varnish/vorigin.vcl
-T 127.0.0.1:6082 -t 120 -w 2000,8000,300 -p thread_pools=2 -p
thread_pool_workspace=100000 -p sess_workspace=1000000 -u varnish -g
varnish -S /etc/varnish/secret -s malloc,90G
Total DISK READ: 0.00 K/s | Total DISK WRITE: 0.00 K/s
PID PRIO USER DISK READ DISK WRITE SWAPIN IO COMMAND
4185 be/4 varnish 0.00 K/s 12659.41 K/s 0.00 % 0.00 % varnishd
-P /var/run/varnish.pid -a :80 -n vorigin -f /etc/varnish/vorigin.vcl
-T 127.0.0.1:6082 -t 120 -w 2000,8000,300 -p thread_pools=2 -p
thread_pool_workspace=100000 -p sess_workspace=1000000 -u varnish -g
varnish -S /etc/varnish/secret -s malloc,90G
I've dug into it with LSOF and sundry, and I can't figure out
what/where varnish is doing the writing. It's not constant... 10-20
seconds will pass before I see it again for several seconds (2-8ish).
NOTE: normal load for these boxes is 3-10k req/sec.
Any thoughts? Why is varnish writing to disk (more to the point:
writing tens megabytes at a time)?
varnish version 3.0.5
Amazon Linux
Kernel 3.14.35-28
Thank you for any help;
-Jason
More information about the varnish-misc
mailing list