<HTML>
<HEAD>
<TITLE>Re: Vanrish 2.1.5 eating memory, hit % decrease</TITLE>
</HEAD>
<BODY>
<FONT SIZE="4"><FONT FACE="Calibri, Verdana, Helvetica, Arial"><SPAN STYLE='font-size:11pt'>Hi Ken,<BR>
<BR>
Thanks for the hint ! <BR>
You’re affecting here 128Mb, how did you get to this munber ? I read somewhere that this value can be set to 10% of the actual memory size which would be in my case 800Mb, does it make sense for you ?<BR>
I read aswell that setting this value to high would crash the system immediately.<BR>
<BR>
<BR>
Yesterday evening, the system was in heavy load but varnish did not hang !<BR>
Instead it dropped all its objects ! Then the load went back fine.<BR>
It seems setting –sfile to 40Gb suits better the memory capability for this server.<BR>
A question remains though ... Why all the objects were dropped ?<BR>
Attached is a plot from cacti regarding the number of objects.<BR>
<BR>
The only thing I could get form the messages log is this :<BR>
Apr 7 19:00:29 server-01-39 varnishd[3732]: Child (3733) died signal=3<BR>
Apr 7 19:00:29 server-01-39 varnishd[3732]: Child cleanup complete<BR>
Apr 7 19:00:29 server-01-39 varnishd[3732]: child (29359) Started<BR>
Apr 7 19:00:29 server-01-39 varnishd[3732]: Child (29359) said <BR>
Apr 7 19:00:29 server-01-39 varnishd[3732]: Child (29359) said Child starts<BR>
Apr 7 19:00:29 server-01-39 varnishd[3732]: Child (29359) said managed to mmap 42949672960 bytes of 42949672960<BR>
<BR>
<BR>
How could I get to know what is realy happening that could explain this behaviour ?<BR>
<BR>
Cheers,<BR>
Jef<BR>
<BR>
Le 08/04/11 01:14, « Ken Brownfield » a écrit :<BR>
<BR>
</SPAN></FONT></FONT><BLOCKQUOTE><FONT SIZE="4"><SPAN STYLE='font-size:11pt'><FONT FACE="Georgia, Times New Roman">This is expected behavior with "-sfile" -- Varnish will use all available RAM to cache its much larger on-disk cache component. If you can get away with memory-only caching, you can use "-smalloc" to utilize a specific amount of RAM.<BR>
<BR>
If you do want to use "-sfile" but are having issue with swap or memory allocation, increasing /proc/sys/vm/min_free_kbytes (or vm.min_free_kbytes via sysctl) to something like 131072 will stabilize performance /in my experience/.<BR>
<BR>
Hope it helps,<BR>
-- <BR>
kb<BR>
</FONT><FONT FACE="Calibri, Verdana, Helvetica, Arial"><BR>
<BR>
<BR>
On Thu, Apr 7, 2011 at 03:38, Jean-Francois Laurens <jean-francois.laurens@rts.ch> wrote:<BR>
</FONT></SPAN></FONT><BLOCKQUOTE><FONT SIZE="4"><SPAN STYLE='font-size:11pt'><FONT FACE="Calibri, Verdana, Helvetica, Arial">Hi there !<BR>
<BR>
I recently updated varnish from version 2.0.6 to 2.1.5 on a centos 5.4 server.<BR>
<BR>
After pretty much a day running 2.1.5, the system went out of memory. The server has 8G of memory and we specified a file of 50Gb as varnish cache file.<BR>
<BR>
I changed some options yesterday following Kristian Lyngstol recommendations ( <FONT COLOR="#134FAE"><U><a href="http://kristianlyng.wordpress.com/2009/10/19/high-end-varnish-tuning/)">http://kristianlyng.wordpress.com/2009/10/19/high-end-varnish-tuning/)</a></U> and did a restart so the system is now running.<BR>
<BR>
I decreased the size of the cache file to 40Gb to.<BR>
<BR>
Here is a top output where you can see that varnish is consuming pretty much all the memory available:<BR>
top -cbn 1<BR>
top - 11:54:24 up 625 days, 19:51, 1 user, load average: 0.52, 0.41, 0.36<BR>
Tasks: 110 total, 1 running, 109 sleeping, 0 stopped, 0 zombie<BR>
Cpu(s): 0.1%us, 0.2%sy, 0.0%ni, 99.1%id, 0.5%wa, 0.0%hi, 0.1%si, 0.0%st<BR>
Mem: 8177764k total, 8132712k used, 45052k free, 13100k buffers<BR>
Swap: 4096564k total, 263324k used, 3833240k free, 7337568k cached<BR>
<BR>
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND <BR>
3733 varnish 15 0 44.6g 7.1g 6.7g S 2.0 91.4 9:56.07 /usr/sbin/varnishd -P /var/run/varnish.pid -a :80 -f /etc/varnish/default.vcl -T 0.0.0.0:6082 <a href="http://0.0.0.0:6082"><http://0.0.0.0:6082></a> -t 120 -w 100,3000,120 -u varnish -g varn<BR>
....<BR>
3732 root 15 0 106m 720 492 S 0.0 0.0 0:00.72 /usr/sbin/varnishd -P /var/run/varnish.pid -a :80 -f /etc/varnish/default.vcl -T 0.0.0.0:6082 <a href="http://0.0.0.0:6082"><http://0.0.0.0:6082></a> -t 120 -w 100,3000,120 -u varnish -g varn<BR>
<BR>
Regularly it’s consuming more and more swap aswell:<BR>
>From 15mn ago:<BR>
free -m<BR>
total used free shared buffers cached<BR>
Mem: 7986 7941 44 0 12 7163<BR>
-/+ buffers/cache: 765 7220<BR>
Swap: 4000 258 3742<BR>
<BR>
>From 5 mn ago:<BR>
free -m<BR>
total used free shared buffers cached<BR>
Mem: 7986 7940 45 0 15 7163<BR>
-/+ buffers/cache: 761 7224<BR>
Swap: 4000 260 3739<BR>
<BR>
>From 1mn ago:<BR>
free -m<BR>
total used free shared buffers cached<BR>
Mem: 7986 7940 45 0 17 7160<BR>
-/+ buffers/cache: 763 7222<BR>
Swap: 4000 262 3737<BR>
<BR>
I understood that varnish relies on the system itself for memory allocation.<BR>
Could we say that the bigger you file cache size is the more varnish need memory ?<BR>
Or Am I facing an issue with my vcl that could have this side effect or ??<BR>
<BR>
In addition to this I've seen the percentage hit decreasing from 65% average with 2.0.6 to 40% with 2.1.5.<BR>
Apart from changing obj to beresp in the vcl_fetch, I had to specify return(deliver) or return(pass) instead of only deliver or pass.<BR>
Would it be a reason for this ?<BR>
Would you be able to give me pointers for me to dig a bit more ?<BR>
<BR>
I Attached a plot from cacti wich would show the difference between 2.0.6 and 2.1.5 (the gap is due to the time to update the server, the trafic was sent to an other server ...)<BR>
</FONT></FONT></SPAN></FONT><FONT FACE="Helvetica, Verdana, Arial"><SPAN STYLE='font-size:9pt'><BR>
</SPAN></FONT><FONT SIZE="4"><FONT FACE="Calibri, Verdana, Helvetica, Arial"><SPAN STYLE='font-size:11pt'><BR>
Thanks in advance for your advices ! <BR>
<BR>
</SPAN></FONT></FONT><FONT COLOR="#808080"><FONT FACE="Arial"><SPAN STYLE='font-size:9pt'>Jean-Francois Laurens<BR>
</SPAN><FONT SIZE="4"><SPAN STYLE='font-size:10pt'>Ingénieur Système Unix<BR>
Resources et Développement<BR>
Secteur Backend<BR>
</SPAN></FONT><SPAN STYLE='font-size:9pt'><B>RTS - Radio Télévision Suisse<BR>
</B>Quai Ernest-Ansermet 20 <BR>
Case postale 234 <BR>
CH - 1211 Genève 8<BR>
T +41 (0)58 236 81 63 <tel:%2B41%20%280%2958%20236%2081%2063> <BR>
</SPAN></FONT><FONT SIZE="4"><FONT FACE="Calibri, Verdana, Helvetica, Arial"><SPAN STYLE='font-size:10pt'><BR>
</SPAN></FONT></FONT></FONT><FONT SIZE="4"><FONT FACE="Calibri, Verdana, Helvetica, Arial"><SPAN STYLE='font-size:11pt'><BR>
_______________________________________________<BR>
varnish-misc mailing list<BR>
varnish-misc@varnish-cache.org<BR>
<a href="http://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc">http://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc</a><BR>
<BR>
<BR>
<HR ALIGN=CENTER SIZE="3" WIDTH="95%"></SPAN></FONT></FONT></BLOCKQUOTE><FONT SIZE="4"><FONT FACE="Consolas, Courier New, Courier"><SPAN STYLE='font-size:10pt'>_______________________________________________<BR>
varnish-misc mailing list<BR>
varnish-misc@varnish-cache.org<BR>
<a href="http://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc">http://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc</a><BR>
</SPAN></FONT></FONT></BLOCKQUOTE><FONT SIZE="4"><FONT FACE="Consolas, Courier New, Courier"><SPAN STYLE='font-size:10pt'><BR>
</SPAN></FONT></FONT><FONT COLOR="#808080"><FONT FACE="Arial"><SPAN STYLE='font-size:9pt'>Jean-Francois Laurens<BR>
</SPAN><FONT SIZE="4"><SPAN STYLE='font-size:10pt'>Ingénieur Système Unix<BR>
Resources et Développement<BR>
Secteur Backend<BR>
</SPAN></FONT><SPAN STYLE='font-size:9pt'><B>RTS - Radio Télévision Suisse<BR>
</B>Quai Ernest-Ansermet 20 <BR>
Case postale 234 <BR>
CH - 1211 Genève 8<BR>
T +41 (0)58 236 81 63<BR>
</SPAN></FONT></FONT><FONT SIZE="4"><FONT FACE="Calibri, Verdana, Helvetica, Arial"><SPAN STYLE='font-size:10pt'><BR>
</SPAN></FONT></FONT>
</BODY>
</HTML>