Re: varnish-dev at varnish-cache.org: Swap flushes all objects in cache

Wed Nov 19 12:48:39 CET 2014

Thank you Per for replying back. Under desperately low memory conditions, the out-of-memory (OOM) killer kicks in and picks up child process of varnish that holds up the virtual memory space (in my case 524785412 kB .i.e 500 GB) and kills it dropping all objects present in the vm space. I am not able to associate the child crash that happens and swapping that place is doing everything wrong here. Can you please re-consider the concerns and have someone look on the errors that are written while breaking.Nov 19 15:06:36 cdn abrt[11775]: Saved core dump of pid 10079 (/usr/sbin/varnishd) to /var/spool/abrt/ccpp-2014-11-19-15:06:33-10079 (381931520 bytes)Nov 19 15:06:36 cdn abrtd: Directory 'ccpp-2014-11-19-15:06:33-10079' creation detectedNov 19 15:06:36 cdn abrtd: Package 'varnish' isn't signed with proper keyNov 19 15:06:36 cdn abrtd: 'post-create' on '/var/spool/abrt/ccpp-2014-11-19-15:06:33-10079' exited with 1Nov 19 15:06:36 cdn abrtd: Deleting problem directory '/var/spool/abrt/ccpp-2014-11-19-15:06:33-10079'Nov 19 15:06:36 cdn varnishd[11921]: Child (10079) not responding to CLI, killing it.Nov 19 15:06:37 cdn varnishd[11921]: Child (10079) died signal=6 (core dumped)Nov 19 15:06:37 cdn varnishd[11921]: Child (10079) Panic message:#012Assert error in cnt_lookup(), cache/cache_req_fsm.c line 411:#012  Condition((oc->exp_flags & (1<<7)) == 0) not true.#012thread = (cache-worker)#012ident =Linux,2.6.32-431.el6.x86_64,x86_64,-sfile,-smalloc,-hclassic,epoll#012Backtrace:#012  0x43b03d: /usr/sbin/varnishd() [0x43b03d]#012  0x43b34d: /usr/sbin/varnishd() [0x43b34d]#012  0x440723: /usr/sbin/varnishd() [0x440723]#012  0x44288d: /usr/sbin/varnishd(CNT_Request+0x441) [0x44288d]#012  0x4339c0: /usr/sbin/varnishd(HTTP1_Session+0x429) [0x4339c0]#012  0x4458cb: /usr/sbin/varnishd() [0x4458cb]#012  0x43dfb5: /usr/sbin/varnishd(Pool_Work_Thread+0x4cb) [0x43dfb5]#012  0x456198: /usr/sbin/varnishd() [0x456198]#012  0x4562c1: /usr/sbin/varnishd(WRK_thread+0x27) [0x4562c1]#012  0x7f4027fc49d1: /lib64/libpthread.so.0(+0x79d1) [0x7f4027fc49d1]#012req = 0x7ec30d0e8020 {#012  sp = 0x7ec30e4257e0, vxid = 1074847970,  step = R_STP_LOOKUP,#012  req_body = R_BODY_NONE,#012  restarts = 0, esi_level = 0#012  sp = 0x7ec30e4257e0 {#012    fd = 75, vxid = 1106140,#012    client = 122.176.68.88 56100,#012    step = S_STP_WORKING,#012  },#012  worker = 0x7f4017709bf0 {#012    ws = 0x7f4017709e08 {#012      id = "wrk",#012      {s,f,r,e} = {0x7f40177093d0,0x7f40177093d0,(nil),+2048},#012    },#012  VCL::method = 0x0,#012  VCL::return = deliver,#012  },#012  ws = 0x7ec30d0e81b8 {#012    id = "req",#012    {s,f,r,e} = {0x7ec30d0ea010,+872,(nil),+57360},#012  },#012  http[req] = {#012    ws = 0x7ec30d0e81b8[req]#012      "GET",#012      "/thumbnail1/3a8bbd2bb61f4f3f.jpg",#012      "HTTP/1.1",#012      "Connection: keep-alive",#012      "Cache-Control: max-age=0",#012      "Accept: image/webp,*/*;q=0.8",#012      "If-Modified-Since: Mon, 08 Sep 2014 03:15:48 GMT",#012      "User-Agent: Mozilla/5.0 (Windows NT 5.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/38.0.2125.111 Safari/537.36",#012      "Referer: http://money.rediff.com/",#012      "Accept-Language: en-US,en;q=0.8",#012      "X-Forwarded-For: 122.176Regards,AnandFrom: Per Buer <perbu at varnish-software.com>Sent: Tue, 18 Nov 2014 23:10:50To: Anand_Shah <anand at rediff-inc.com>Subject: Re: varnish-dev at varnish-cache.org: Swap flushes all objects in cache Anand, On Tue, Nov 18, 2014 at 2:01 PM, Anand Shah <anand at rediff-inc.com> wrote:
I have noticed this everytime the memory offload happens. This is not a crash or segfault; coz the service does not go down and it keeps in running which is also evident in the graphs shared earlier. I can see the same behaviour everytime when operating system swaps. Can you confirm if this is a ideal behaviour with varnish? 

Please keep user questions to the -misc list.

This is most like misconfiguration causing the OOM killer to kill of the child process. It doesn't keep running. Look at the accepted sessions in the graph. It dives when this happens.

 -- 
									Per Buer			CTO | Varnish Software AS			Cell: +47 95839117			We Make Websites Fly!			www.varnish-software.com			 
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/varnish-dev/attachments/20141119/7e55a53e/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: nstruct1.jpg
Type: image/jpeg
Size: 344460 bytes
Desc: not available
URL: <https://www.varnish-cache.org/lists/pipermail/varnish-dev/attachments/20141119/7e55a53e/attachment-0002.jpg>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: pageout1.jpg
Type: image/jpeg
Size: 338673 bytes
Desc: not available
URL: <https://www.varnish-cache.org/lists/pipermail/varnish-dev/attachments/20141119/7e55a53e/attachment-0003.jpg>