Vanrish 2.1.5 eating memory, hit % decrease
Jean-Francois Laurens
jean-francois.laurens at rts.ch
Fri Apr 8 10:55:10 CEST 2011
Hi Ken,
Thanks for the hint !
You¹re affecting here 128Mb, how did you get to this munber ? I read
somewhere that this value can be set to 10% of the actual memory size which
would be in my case 800Mb, does it make sense for you ?
I read aswell that setting this value to high would crash the system
immediately.
Yesterday evening, the system was in heavy load but varnish did not hang !
Instead it dropped all its objects ! Then the load went back fine.
It seems setting sfile to 40Gb suits better the memory capability for this
server.
A question remains though ... Why all the objects were dropped ?
Attached is a plot from cacti regarding the number of objects.
The only thing I could get form the messages log is this :
Apr 7 19:00:29 server-01-39 varnishd[3732]: Child (3733) died signal=3
Apr 7 19:00:29 server-01-39 varnishd[3732]: Child cleanup complete
Apr 7 19:00:29 server-01-39 varnishd[3732]: child (29359) Started
Apr 7 19:00:29 server-01-39 varnishd[3732]: Child (29359) said
Apr 7 19:00:29 server-01-39 varnishd[3732]: Child (29359) said Child starts
Apr 7 19:00:29 server-01-39 varnishd[3732]: Child (29359) said managed to
mmap 42949672960 bytes of 42949672960
How could I get to know what is realy happening that could explain this
behaviour ?
Cheers,
Jef
Le 08/04/11 01:14, « Ken Brownfield » a écrit :
> This is expected behavior with "-sfile" -- Varnish will use all available RAM
> to cache its much larger on-disk cache component. If you can get away with
> memory-only caching, you can use "-smalloc" to utilize a specific amount of
> RAM.
>
> If you do want to use "-sfile" but are having issue with swap or memory
> allocation, increasing /proc/sys/vm/min_free_kbytes (or vm.min_free_kbytes via
> sysctl) to something like 131072 will stabilize performance /in my
> experience/.
>
> Hope it helps,
> --
> kb
>
>
>
> On Thu, Apr 7, 2011 at 03:38, Jean-Francois Laurens
> <jean-francois.laurens at rts.ch> wrote:
>> Hi there !
>>
>> I recently updated varnish from version 2.0.6 to 2.1.5 on a centos 5.4
>> server.
>>
>> After pretty much a day running 2.1.5, the system went out of memory. The
>> server has 8G of memory and we specified a file of 50Gb as varnish cache
>> file.
>>
>> I changed some options yesterday following Kristian Lyngstol recommendations
>> ( http://kristianlyng.wordpress.com/2009/10/19/high-end-varnish-tuning/) and
>> did a restart so the system is now running.
>>
>> I decreased the size of the cache file to 40Gb to.
>>
>> Here is a top output where you can see that varnish is consuming pretty much
>> all the memory available:
>> top -cbn 1
>> top - 11:54:24 up 625 days, 19:51, 1 user, load average: 0.52, 0.41, 0.36
>> Tasks: 110 total, 1 running, 109 sleeping, 0 stopped, 0 zombie
>> Cpu(s): 0.1%us, 0.2%sy, 0.0%ni, 99.1%id, 0.5%wa, 0.0%hi, 0.1%si,
>> 0.0%st
>> Mem: 8177764k total, 8132712k used, 45052k free, 13100k buffers
>> Swap: 4096564k total, 263324k used, 3833240k free, 7337568k cached
>>
>> PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
>>
>>
>> 3733 varnish 15 0 44.6g 7.1g 6.7g S 2.0 91.4 9:56.07
>> /usr/sbin/varnishd -P /var/run/varnish.pid -a :80 -f /etc/varnish/default.vcl
>> -T 0.0.0.0:6082 <http://0.0.0.0:6082> -t 120 -w 100,3000,120 -u varnish -g
>> varn
>> ....
>> 3732 root 15 0 106m 720 492 S 0.0 0.0 0:00.72
>> /usr/sbin/varnishd -P /var/run/varnish.pid -a :80 -f /etc/varnish/default.vcl
>> -T 0.0.0.0:6082 <http://0.0.0.0:6082> -t 120 -w 100,3000,120 -u varnish -g
>> varn
>>
>> Regularly it¹s consuming more and more swap aswell:
>> From 15mn ago:
>> free -m
>> total used free shared buffers cached
>> Mem: 7986 7941 44 0 12 7163
>> -/+ buffers/cache: 765 7220
>> Swap: 4000 258 3742
>>
>> From 5 mn ago:
>> free -m
>> total used free shared buffers cached
>> Mem: 7986 7940 45 0 15 7163
>> -/+ buffers/cache: 761 7224
>> Swap: 4000 260 3739
>>
>> From 1mn ago:
>> free -m
>> total used free shared buffers cached
>> Mem: 7986 7940 45 0 17 7160
>> -/+ buffers/cache: 763 7222
>> Swap: 4000 262 3737
>>
>> I understood that varnish relies on the system itself for memory allocation.
>> Could we say that the bigger you file cache size is the more varnish need
>> memory ?
>> Or Am I facing an issue with my vcl that could have this side effect or ??
>>
>> In addition to this I've seen the percentage hit decreasing from 65% average
>> with 2.0.6 to 40% with 2.1.5.
>> Apart from changing obj to beresp in the vcl_fetch, I had to specify
>> return(deliver) or return(pass) instead of only deliver or pass.
>> Would it be a reason for this ?
>> Would you be able to give me pointers for me to dig a bit more ?
>>
>> I Attached a plot from cacti wich would show the difference between 2.0.6 and
>> 2.1.5 (the gap is due to the time to update the server, the trafic was sent
>> to an other server ...)
>>
>>
>> Thanks in advance for your advices !
>>
>> Jean-Francois Laurens
>> Ingénieur Système Unix
>> Resources et Développement
>> Secteur Backend
>> RTS - Radio Télévision Suisse
>> Quai Ernest-Ansermet 20
>> Case postale 234
>> CH - 1211 Genève 8
>> T +41 (0)58 236 81 63 <tel:%2B41%20%280%2958%20236%2081%2063>
>>
>>
>> _______________________________________________
>> varnish-misc mailing list
>> varnish-misc at varnish-cache.org
>> http://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
>>
>>
>>
>> _______________________________________________
>> varnish-misc mailing list
>> varnish-misc at varnish-cache.org
>> http://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
>>
>> Jean-Francois Laurens
>> Ingénieur Système Unix
>> Resources et Développement
>> Secteur Backend
>> RTS - Radio Télévision Suisse
>> Quai Ernest-Ansermet 20
>> Case postale 234
>> CH - 1211 Genève 8
>> T +41 (0)58 236 81 63
>>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/varnish-misc/attachments/20110408/6876456f/attachment-0003.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: varnish_data_structure_size_cacti.png
Type: application/octet-stream
Size: 52759 bytes
Desc: not available
URL: <https://www.varnish-cache.org/lists/pipermail/varnish-misc/attachments/20110408/6876456f/attachment-0003.obj>
More information about the varnish-misc
mailing list