varnish-dev at varnish-cache.org: Swap flushes all objects in cache

[Federico Schwindt]

Hi,

This looks like https://www.varnish-cache.org/trac/ticket/1539.

What varnish version are you running?

On Wed, Nov 19, 2014 at 11:48 AM, Anand Shah <anand at rediff-inc.com> wrote:

>
>
> Thank you Per for replying back.
>
>
> Under desperately low memory conditions, the out-of-memory (OOM) killer
> kicks in and picks up child process of varnish that holds up the virtual
> memory space (in my case 524785412 kB .i.e 500 GB) and kills it dropping
> all objects present in the vm space. I am not able to associate the child
> crash that happens and swapping that place is doing everything wrong here.
> Can you please re-consider the concerns and have someone look on the errors
> that are written while breaking.
>
>
>
> Nov 19 15:06:36 cdn abrt[11775]: Saved core dump of pid 10079
> (/usr/sbin/varnishd) to /var/spool/abrt/ccpp-2014-11-19-15:06:33-10079
> (381931520 bytes)
> Nov 19 15:06:36 cdn abrtd: Directory 'ccpp-2014-11-19-15:06:33-10079'
> creation detected
> Nov 19 15:06:36 cdn abrtd: Package 'varnish' isn't signed with proper key
> Nov 19 15:06:36 cdn abrtd: 'post-create' on
> '/var/spool/abrt/ccpp-2014-11-19-15:06:33-10079' exited with 1
> Nov 19 15:06:36 cdn abrtd: Deleting problem directory
> '/var/spool/abrt/ccpp-2014-11-19-15:06:33-10079'
> Nov 19 15:06:36 cdn varnishd[11921]: Child (10079) not responding to CLI,
> killing it.
> Nov 19 15:06:37 cdn varnishd[11921]: Child (10079) died signal=6 (core
> dumped)
>
> Nov 19 15:06:37 cdn varnishd[11921]: Child (10079) Panic
> message:#012Assert error in cnt_lookup(), cache/cache_req_fsm.c line
> 411:#012  Condition((oc->exp_flags & (1<<7)) == 0) not true.#012thread =
> (cache-worker)#012ident =
> Linux,2.6.32-431.el6.x86_64,x86_64,-sfile,-smalloc,-hclassic,epoll#012Backtrace:#012
>  0x43b03d: /usr/sbin/varnishd() [0x43b03d]#012  0x43b34d:
> /usr/sbin/varnishd() [0x43b34d]#012  0x440723: /usr/sbin/varnishd()
> [0x440723]#012  0x44288d: /usr/sbin/varnishd(CNT_Request+0x441)
> [0x44288d]#012  0x4339c0: /usr/sbin/varnishd(HTTP1_Session+0x429)
> [0x4339c0]#012  0x4458cb: /usr/sbin/varnishd() [0x4458cb]#012  0x43dfb5:
> /usr/sbin/varnishd(Pool_Work_Thread+0x4cb) [0x43dfb5]#012  0x456198:
> /usr/sbin/varnishd() [0x456198]#012  0x4562c1:
> /usr/sbin/varnishd(WRK_thread+0x27) [0x4562c1]#012  0x7f4027fc49d1:
> /lib64/libpthread.so.0(+0x79d1) [0x7f4027fc49d1]#012req = 0x7ec30d0e8020
> {#012  sp = 0x7ec30e4257e0, vxid = 1074847970,  step = R_STP_LOOKUP,#012
>  req_body = R_BODY_NONE,#012  restarts = 0, esi_level = 0#012  sp =
> 0x7ec30e4257e0 {#012    fd = 75, vxid = 1106140,#012    client =
> 122.176.68.88 56100,#012    step = S_STP_WORKING,#012  },#012  worker =
> 0x7f4017709bf0 {#012    ws = 0x7f4017709e08 {#012      id = "wrk",#012
>  {s,f,r,e} = {0x7f40177093d0,0x7f40177093d0,(nil),+2048},#012    },#012
>  VCL::method = 0x0,#012  VCL::return = deliver,#012  },#012  ws =
> 0x7ec30d0e81b8 {#012    id = "req",#012    {s,f,r,e} =
> {0x7ec30d0ea010,+872,(nil),+57360},#012  },#012  http[req] = {#012    ws =
> 0x7ec30d0e81b8[req]#012      "GET",#012
>  "/thumbnail1/3a8bbd2bb61f4f3f.jpg",#012      "HTTP/1.1",#012
>  "Connection: keep-alive",#012      "Cache-Control: max-age=0",#012
>  "Accept: image/webp,*/*;q=0.8",#012      "If-Modified-Since: Mon, 08 Sep
> 2014 03:15:48 GMT",#012      "User-Agent: Mozilla/5.0 (Windows NT 5.1)
> AppleWebKit/537.36 (KHTML, like Gecko) Chrome/38.0.2125.111
> Safari/537.36",#012      "Referer: http://money.rediff.com/",#012
>  "Accept-Language: en-US,en;q=0.8",#012      "X-Forwarded-For: 122.176
>
>
>
> Regards,
> Anand
>
>
>
> From: Per Buer <perbu at varnish-software.com>
> Sent: Tue, 18 Nov 2014 23:10:50
> To: Anand_Shah <anand at rediff-inc.com>
> Subject: Re: varnish-dev at varnish-cache.org: Swap flushes all objects in
> cache
>
>
> Anand,
>
> On Tue, Nov 18, 2014 at 2:01 PM, Anand Shah <anand at rediff-inc.com> wrote:
>>
>> I have noticed this everytime the memory offload happens. This is not a
>> crash or segfault; coz the service does not go down and it keeps in running
>> which is also evident in the graphs shared earlier. I can see the same
>> behaviour everytime when operating system swaps. Can you confirm if this is
>> a ideal behaviour with varnish?
>
>
> Please keep user questions to the -misc list.
>
> This is most like misconfiguration causing the OOM killer to kill of the
> child process. It doesn't keep running. Look at the accepted sessions in
> the graph. It dives when this happens.
>
>
> --
>   *Per Buer*
> CTO | Varnish Software AS
> Cell: +47 95839117
> We Make Websites Fly!
> www.varnish-software.com
> [image: Register now]
> <http://info.varnish-software.com/varnish-summits-autumn-2014-registration>
>
> _______________________________________________
> varnish-dev mailing list
> varnish-dev at varnish-cache.org
> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/varnish-dev/attachments/20141125/9d9b8162/attachment.html>