<HTML>
<HEAD>
<TITLE>Re: Vanrish 2.1.5 eating memory, hit % decrease</TITLE>
</HEAD>
<BODY>
<FONT SIZE="4"><FONT FACE="Calibri, Verdana, Helvetica, Arial"><SPAN STYLE='font-size:11pt'>Hi Ken, others,<BR>
<BR>
Just a feedback for the record about persistent option; the varnish process failed at some point with the following error in the logs:<BR>
Apr 15 00:16:47 server-01-39 varnishtsr[3726]: Child (13026) Panic message: Assert error in smp_open_segs(), storage_persistent.c line 1026: Condition(sg1->p.offset != sg->p.offset) not true. errno = 9 (Bad file descriptor) thread = (cache-main) ident = Linux,2.6.18-194.32.1.el5,x86_64,-spersistent,-hcritbit,no_waiter Backtrace: 0x424446: /usr/sbin/varnishd [0x424446] 0x43e505: /usr/sbin/varnishd [0x43e505] 0x43e6eb: /usr/sbin/varnishd [0x43e6eb] 0x439abe: /usr/sbin/varnishd(STV_open+0x1e) [0x439abe] 0x4234ef: /usr/sbin/varnishd(child_main+0xbf) [0x4234ef] 0x432630: /usr/sbin/varnishd [0x432630] 0x432e59: /usr/sbin/varnishd [0x432e59] 0x39316084f7: /usr/lib64/libvarnish.so.1 [0x39316084f7] 0x3931608b88: /usr/lib64/libvarnish.so.1(vev_schedule+0x88) [0x3931608b88] 0x432893: /usr/sbin/varnishd(MGT_Run+0x143) [0x432893] <BR>
<BR>
I just stoped using the persistent cache as I’m just unable to understand and investigate the root cause of the problem ( where is this “. errno = 9 (Bad file descriptor) “ error coming from ?) <BR>
Using it for production seems to me just not reasonable at the moment. Certainly a version 3 will handle it properly !<BR>
<BR>
Nevertheless your suggestion about setting the vm.min_free_kbytes did the trick, I guess.<BR>
I’m testing it right now with 64M and see over the time if the system remains stable.<BR>
<BR>
What I see now is that the load remains pretty equal no matter how heavy the trafic is.<BR>
The number of objects seems to stay stable, meaning no child process get killed and objects lost.<BR>
<BR>
<IMG src="cid:3386164671_4978398" ><IMG src="cid:3386164671_4954354" ><IMG src="cid:3386164671_4997738" ><IMG src="cid:3386164671_5002531" ><IMG src="cid:3386164671_4948071" ><BR>
<BR>
Le 08/04/11 22:55, « Ken Brownfield » a écrit :<BR>
<BR>
</SPAN></FONT></FONT><BLOCKQUOTE><FONT SIZE="4"><SPAN STYLE='font-size:11pt'><FONT FACE="Georgia, Times New Roman">This means the child process died and restarted (the reason for this should appear earlier in the log; perhaps your cli_timeout is too low under a heavily loaded system -- try 20s).<BR>
<BR>
"-sfile" is not persistent storage, so when the child process restarts it uses a new, empty storage structure. You should have luck with "-spersistent" on the latest Varnish or trunk, at least for child process restarts.<BR>
<BR>
FWIW,<BR>
-- <BR>
kb<BR>
</FONT><FONT FACE="Calibri, Verdana, Helvetica, Arial"><BR>
<BR>
<BR>
On Fri, Apr 8, 2011 at 01:55, Jean-Francois Laurens <jean-francois.laurens@rts.ch> wrote:<BR>
</FONT></SPAN></FONT><BLOCKQUOTE><FONT SIZE="4"><SPAN STYLE='font-size:11pt'><FONT FACE="Calibri, Verdana, Helvetica, Arial">Hi Ken,<BR>
<BR>
Thanks for the hint ! <BR>
You’re affecting here 128Mb, how did you get to this munber ? I read somewhere that this value can be set to 10% of the actual memory size which would be in my case 800Mb, does it make sense for you ?<BR>
I read aswell that setting this value to high would crash the system immediately.<BR>
<BR>
<BR>
Yesterday evening, the system was in heavy load but varnish did not hang !<BR>
Instead it dropped all its objects ! Then the load went back fine.<BR>
It seems setting –sfile to 40Gb suits better the memory capability for this server.<BR>
A question remains though ... Why all the objects were dropped ?<BR>
Attached is a plot from cacti regarding the number of objects.<BR>
<BR>
The only thing I could get form the messages log is this :<BR>
Apr 7 19:00:29 server-01-39 varnishd[3732]: Child (3733) died signal=3<BR>
Apr 7 19:00:29 server-01-39 varnishd[3732]: Child cleanup complete<BR>
Apr 7 19:00:29 server-01-39 varnishd[3732]: child (29359) Started<BR>
Apr 7 19:00:29 server-01-39 varnishd[3732]: Child (29359) said <BR>
Apr 7 19:00:29 server-01-39 varnishd[3732]: Child (29359) said Child starts<BR>
Apr 7 19:00:29 server-01-39 varnishd[3732]: Child (29359) said managed to mmap 42949672960 bytes of 42949672960<BR>
<BR>
<BR>
How could I get to know what is realy happening that could explain this behaviour ?<BR>
<BR>
Cheers,<BR>
Jef<BR>
</FONT></SPAN></FONT></BLOCKQUOTE><FONT SIZE="4"><SPAN STYLE='font-size:11pt'><FONT FACE="Calibri, Verdana, Helvetica, Arial"><BR>
<HR ALIGN=CENTER SIZE="3" WIDTH="95%"></FONT></SPAN><FONT FACE="Consolas, Courier New, Courier"><SPAN STYLE='font-size:10pt'>_______________________________________________<BR>
varnish-misc mailing list<BR>
varnish-misc@varnish-cache.org<BR>
<a href="http://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc">http://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc</a><BR>
</SPAN></FONT></FONT></BLOCKQUOTE><FONT SIZE="4"><FONT FACE="Consolas, Courier New, Courier"><SPAN STYLE='font-size:10pt'><BR>
</SPAN></FONT></FONT><FONT COLOR="#808080"><FONT FACE="Arial"><SPAN STYLE='font-size:9pt'>Jean-Francois Laurens<BR>
</SPAN><FONT SIZE="4"><SPAN STYLE='font-size:10pt'>Ingénieur Système Unix<BR>
Resources et Développement<BR>
Secteur Backend<BR>
</SPAN></FONT><SPAN STYLE='font-size:9pt'><B>RTS - Radio Télévision Suisse<BR>
</B>Quai Ernest-Ansermet 20 <BR>
Case postale 234 <BR>
CH - 1211 Genève 8<BR>
T +41 (0)58 236 81 63<BR>
</SPAN></FONT></FONT><FONT SIZE="4"><FONT FACE="Calibri, Verdana, Helvetica, Arial"><SPAN STYLE='font-size:10pt'><BR>
</SPAN></FONT></FONT>
</BODY>
</HTML>