varnish nuking agressively all my objects.

[Aurélien Lemaire]

On 27/09/11 12:43, Aurélien Lemaire wrote:
>
> Le 16/09/2011 08:55, Jorge Nerín a écrit :
>>
>> On Thu, Sep 15, 2011 at 11:29, Aurélien Lemaire <aurelien.lemaire at smile.fr 
>> <mailto:aurelien.lemaire at smile.fr>> wrote:
>>
>>     Good day folks,
>>
>>     First of all, varnish is an outstanding piece of software that my company
>>     and i are addicted to. So thanks to all the coders.
>>
>>     Here is my problem :
>>     I allocated varnish 1G of RAM on a website that can have more than 2 Go
>>     of possible cacheable objects . Not to worry though as any proxy-cache
>>     system should smartly nuke old objects to make place to new one to live
>>     peacefully within its allocated RAM. And that's where Varnish behave
>>     unexpectedly : each time it need to nuke SOME objects : it nukes ALMOST
>>     ALL of them (often ~80% of my 35k objects) which is quite aggressive ;
>>     thus i lost almost all my cache....IRK !
>>
>>     3 Munin graphs attached to see the problem clearly : big drop each time a
>>     nuking happens.
>>
>>     To make sure my pbr is about varnish nuking system : i increased from 1G
>>     to 3G(more than the max possible 2G cacheable objects) on another varnish
>>     of this platefom (this website is delivered by multiple front/varnish
>>     server all stricly similar and independant) and this issue disappeared
>>     (no more nuking : no lost of ~80%of my objects)
>>
>>     Here is my env :
>>     Debian 5.0.8 64 bits on 2.6.32-5-openvz-amd64 kernel
>>     Varnish 2.1.3 SVN 5049:5055(debian package 2.1.3-8)
>>     200 varnish 's worker threads running constantly (no issue on workers)
>>     30req/s average with 60/s in peak
>>
>>     Daemon run as such :
>>     /usr/sbin/varnishd -P /var/run/varnishd.pid -a :80 -T localhost:6082 -S
>>     /etc/varnish/secret -f /etc/varnish/serverx.vcl -w 100,1024 -s
>>     file,/var/lib/varnish/serverx/varnish_storage.bin,3G
>>
>>     Here a quick varnishstat -1 :
>>
>>     Is it normal varnish behaviour ? sounds like a bug to me.
>>     Am i missing some tuning (lru_interval)  to soften the nuking algo ?
>>     Do you need more info ?
>>     helps appreciated here  ;-)
>>
>>     Regards, Aurelien Lemaire
>>
>>
>> It could be someone downloading a large file (like a ~700Mb iso file) and 
>> varnish nuking objects to make room for this file (even if its configured to 
>> not cache it).
>>
>> Try to get a varnishlog trace of the moment the nuking begins.
>>
>> -- 
>> Jorge Nerín
>> <jnerin at gmail.com <mailto:jnerin at gmail.com>>
> This website does not content any files bigger than a couple of Mb.
>
> Will see what i can do to get the varnishlog during the nuking.
>
> --
> aurelien lemaire

Providing a reduced-to-the-minimum varnishlog on this stressed prod env won't be 
easy for me moreover i doubt that nuking activity is reported by varnishlog . So 
meanwhile :

My "agressive nuking" problem seems to be known and often-encountered  :
- http://www.gossamer-threads.com/lists/varnish/dev/9631
- http://www.mail-archive.com/varnish-misc@projects.linpro.no/msg03684.html
- Quite some thread without any outcome found by google search where all the 
people just fix it by increasign the the "-s file" size which only postpone the 
problem

On this first link Artur ( a varnish dev 
http://www.gossamer-threads.com/lists/engine?user=137;list=varnish ) shortly 
admit "/... There are problems with the fragmentation of the store.../" on the 
1st of april 2009.
On the second link there is all the varnishlog and backtrace infos needed to 
open the bug report to get it fixed

Thus is it possible to open this bug report officially as "to be fixed" ?

At the same time i'll try to swith  to "-s malloc " as suggested by Artur and 
keep you all informed.

Cheers,
--
Aurélien Lemaire

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/varnish-misc/attachments/20110928/1589f507/attachment-0003.html>