Vanrish 2.1.5 eating memory, hit % decrease

Wed Apr 20 17:17:51 CEST 2011

Hi Ken, others,

Just a feedback for the record about persistent option; the varnish process
failed at some point with the following error in the logs:
Apr 15 00:16:47 server-01-39 varnishtsr[3726]: Child (13026) Panic message:
Assert error in smp_open_segs(), storage_persistent.c line 1026:
Condition(sg1->p.offset != sg->p.offset) not true. errno = 9 (Bad file
descriptor) thread = (cache-main) ident =
Linux,2.6.18-194.32.1.el5,x86_64,-spersistent,-hcritbit,no_waiter Backtrace:
0x424446: /usr/sbin/varnishd [0x424446]   0x43e505: /usr/sbin/varnishd
[0x43e505]   0x43e6eb: /usr/sbin/varnishd [0x43e6eb]   0x439abe:
/usr/sbin/varnishd(STV_open+0x1e) [0x439abe]   0x4234ef:
/usr/sbin/varnishd(child_main+0xbf) [0x4234ef]   0x432630:
/usr/sbin/varnishd [0x432630]   0x432e59: /usr/sbin/varnishd [0x432e59]
0x39316084f7: /usr/lib64/libvarnish.so.1 [0x39316084f7]   0x3931608b88:
/usr/lib64/libvarnish.so.1(vev_schedule+0x88) [0x3931608b88]   0x432893:
/usr/sbin/varnishd(MGT_Run+0x143) [0x432893]

I just stoped using the persistent cache as I¹m just unable to understand
and investigate the root cause of the problem ( where is this ³. errno = 9
(Bad file descriptor) ³ error coming from ?)
Using it for production seems to me just not reasonable at the moment.
Certainly a version 3 will handle it properly !

 Nevertheless your suggestion about setting the vm.min_free_kbytes did the
trick, I guess.
I¹m testing it right now with 64M and see over the time if the system
remains stable.

What I see now is that the load remains pretty equal no matter how heavy the
trafic is.
The number of objects seems to stay stable, meaning no child process get
killed and objects lost.

Le 08/04/11 22:55, « Ken Brownfield » a écrit :

> This means the child process died and restarted (the reason for this should
> appear earlier in the log; perhaps your cli_timeout is too low under a heavily
> loaded system -- try 20s).
> 
> "-sfile" is not persistent storage, so when the child process restarts it uses
> a new, empty storage structure.  You should have luck with "-spersistent" on
> the latest Varnish or trunk, at least for child process restarts.
> 
> FWIW,
> -- 
> kb
> 
> 
> 
> On Fri, Apr 8, 2011 at 01:55, Jean-Francois Laurens
> <jean-francois.laurens at rts.ch> wrote:
>> Hi Ken,
>> 
>> Thanks for the hint !
>> You¹re affecting here 128Mb, how did you get to this munber ? I read
>> somewhere that this value can be set to 10% of the actual memory size which
>> would be in my case 800Mb, does it make sense for you ?
>> I read aswell that setting this value to high would crash the system
>> immediately.
>> 
>> 
>> Yesterday evening, the system was in heavy load but varnish did not hang !
>> Instead it dropped all its objects ! Then the load went back fine.
>> It seems setting sfile to 40Gb suits better the memory capability for this
>> server.
>> A question remains though ... Why all the objects were dropped ?
>> Attached is a plot from cacti regarding the number of objects.
>> 
>> The only thing I could get form the messages log is this :
>> Apr  7 19:00:29 server-01-39 varnishd[3732]: Child (3733) died signal=3
>> Apr  7 19:00:29 server-01-39 varnishd[3732]: Child cleanup complete
>> Apr  7 19:00:29 server-01-39 varnishd[3732]: child (29359) Started
>> Apr  7 19:00:29 server-01-39 varnishd[3732]: Child (29359) said
>> Apr  7 19:00:29 server-01-39 varnishd[3732]: Child (29359) said Child starts
>> Apr  7 19:00:29 server-01-39 varnishd[3732]: Child (29359) said managed to
>> mmap 42949672960 bytes of 42949672960
>> 
>> 
>> How could I get to know what is realy happening that could explain this
>> behaviour ?
>> 
>> Cheers,
>> Jef
> 
> 
> _______________________________________________
> varnish-misc mailing list
> varnish-misc at varnish-cache.org
> http://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
> 
> Jean-Francois Laurens
> Ingénieur Système Unix
> Resources et Développement
> Secteur Backend
> RTS - Radio Télévision Suisse
> Quai Ernest-Ansermet 20
> Case postale 234 
> CH - 1211 Genève 8
> T +41 (0)58 236 81 63
> 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/varnish-misc/attachments/20110420/7def90d0/attachment-0003.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image.png
Type: image/png
Size: 50979 bytes
Desc: not available
URL: <https://www.varnish-cache.org/lists/pipermail/varnish-misc/attachments/20110420/7def90d0/attachment-0015.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image.png
Type: image/png
Size: 29315 bytes
Desc: not available
URL: <https://www.varnish-cache.org/lists/pipermail/varnish-misc/attachments/20110420/7def90d0/attachment-0016.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image.png
Type: image/png
Size: 31669 bytes
Desc: not available
URL: <https://www.varnish-cache.org/lists/pipermail/varnish-misc/attachments/20110420/7def90d0/attachment-0017.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image.png
Type: image/png
Size: 33211 bytes
Desc: not available
URL: <https://www.varnish-cache.org/lists/pipermail/varnish-misc/attachments/20110420/7def90d0/attachment-0018.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image.png
Type: image/png
Size: 38491 bytes
Desc: not available
URL: <https://www.varnish-cache.org/lists/pipermail/varnish-misc/attachments/20110420/7def90d0/attachment-0019.png>