varnish core dump

Mahesh Ollalwar mahesho at outlook.com
Wed Sep 14 07:14:53 CEST 2016


Thank you Anand.

we have set workspace_backend=262144 and the issue is resolved.


Thanks,

Mahesh.

________________________________
From: anand at rediff-inc.com <anand at rediff-inc.com> on behalf of Anand Shah <anand at rediff-inc.com>
Sent: Monday, September 12, 2016 4:09 PM
To: Mahesh Ollalwar
Cc: Dag Haavi Finstad ; varnish-misc at varnish-cache.org
Subject: Re: varnish core dump

In order to reduce contention between all the threads, Varnish uses workspaces which allow each thread to use or modify memory independently of other threads, which avoids locking. Try increasing session workspace and shm_workspace. Disable any VMOD's that you use.



Regards,
Anand Shah



From: Mahesh Ollalwar <mahesho at outlook.com>
Sent: Mon, 12 Sep 2016 15:16:52
To: Dag Haavi Finstad <daghf at varnish-software.com>, "varnish-misc at varnish-cache.org" <varnish-misc at varnish-cache.org>
Subject: Re: varnish core dump




Hi,



Thanks for the response.

We have upgraded to latest stable version 4.1.3 but still see panics. below are the logs.





Sep 12 12:45:18 Prod-SHARED-CACHE6 abrt[32688]: Saved core dump of pid 24551 (/product/varnish/sbin/varnishd) to /var/spool/abrt/ccpp-2016-09-12-12:45:15-24551 (585650176 bytes)
Sep 12 12:45:18 Prod-SHARED-CACHE6 abrtd: Directory 'ccpp-2016-09-12-12:45:15-24551' creation detected
Sep 12 12:45:19 Prod-SHARED-CACHE6 abrtd: Executable '/product/varnish/sbin/varnishd' doesn't belong to any package and ProcessUnpackaged is set to 'no'
Sep 12 12:45:19 Prod-SHARED-CACHE6 abrtd: 'post-create' on '/var/spool/abrt/ccpp-2016-09-12-12:45:15-24551' exited with 1
Sep 12 12:45:19 Prod-SHARED-CACHE6 abrtd: Deleting problem directory '/var/spool/abrt/ccpp-2016-09-12-12:45:15-24551'
Sep 12 12:45:19 Prod-SHARED-CACHE6 varnishd[11105]: Child (24551) not responding to CLI, killing it.
Sep 12 12:45:20 Prod-SHARED-CACHE6 varnishd[11105]: Child (24551) died signal=6 (core dumped)
Sep 12 12:45:20 Prod-SHARED-CACHE6 varnishd[11105]: Child (24551) Last panic at: Mon, 12 Sep 2016 07:15:20 GMT#012"Assert error in http_EstimateWS(), cache/cache_http.c line 793:#012  Condition((fm->hd[u].b) != 0) not true.#012thread = (cache-worker)#012version = varnish-4.1.3 revision 5e3b6d2#012ident = Linux,2.6.32-431.11.2.el6.x86_64,x86_64,-junix,-smalloc,-smalloc,-hcritbit,epoll#012Backtrace:#012  0x432f13: pan_ic+0x163#012  0x42dec5: http_EstimateWS+0x435#012  0x421874: vbf_beresp2obj+0x84#012  0x422cb5: vbf_stp_error+0x235#012  0x42366b: vbf_fetch_thread+0x1db#012  0x449825: WRK_Thread+0x4a5#012  0x449c5b: pool_thread+0x2b#012  0x3c25e079d1: libpthread.so<http://libpthread.so>.0() [0x3c25e079d1]#012  0x3c25ae8b6d: libc.so<http://libc.so>.6(clone+0x6d) [0x3c25ae8b6d]#012busyobj = 0x7f7f505208a0 {#012  ws = 0x7f7f50520960 {#012    OVERFLOWED id = \"Bo\",#012    {s,f,r,e} = {0x7f7f50522818,+57440,(nil),+57440},#012  },#012  refcnt = 2,#012  retries = 0, failed = 1, state = 1,#012  flags = {do_gzip, do_stream, is_gunzip},#012  director_req = 0x1a16130 {#012    vcl_name = farm_anon,#012    type = round-robin {#012    },#012  },#012  director_resp = 0x1a08e98 {#012    vcl_name = PROD_LB_BE_ANON_1,#012    type = backend {#012      display_name = boot.PROD_LB_BE_ANON_1,#012      ipv4 = 10.254.4.254<http://10.254.4.254>,#012      port = 8080,#012      hosthdr = 10.254.4.254<http://10.254.4.254>,#012      health=healthy, admin_health=probe, changed=1473664247.5,#012      n_conn = 2,#012    },#012  },#012  http[bereq] = 0x7f7f50520f28 {#012    ws[Bo] = 0x7f7f50520960,#012    hdrs {#012      \"GET\",#012      \"/AVS/besc?action=GetArrayContentList&channel=ANDROID&categoryId=7022\",#012      \"HTTP/1.1\",#012      \"host: www.example.com<http://www.example.com> \",#012      \"Accept-Encoding: gzip\",#012      \"Akamai-Origin-Hop: 4\",#012      \"True-Client-IP: 183.83.249.187<http://183.83.249.187>\",#012      \"User-Agent: Dalvik/2.1.0 (Linux; U; Android 6.0.1; Le X526 Build/IIXOSOP5801607291S)\",#012      \"Via: 1.1 v1-akamaitech.net<http://v1-akamaitech.net>(ghost) (AkamaiGHost), 1.1 v1-akamaitech.net<http://v1-akamaitech.net>(ghost) (AkamaiGHost), 1.1 v1-akamaitech.net<http://v1-akamaitech.net>(ghost) (AkamaiGHost), 1.1 aka
Sep 12 12:45:20 Prod-SHARED-CACHE6 varnishd[11105]: Child (386) said Child starts





Thanks in advance,

Mahesh.


________________________________
From: Dag Haavi Finstad <daghf at varnish-software.com>
Sent: Monday, June 27, 2016 2:57 PM
To: Mahesh Ollalwar
Cc: varnish-misc at varnish-cache.org
Subject: Re: varnish core dump

Hi

I believe this might be #1539 [1], which is fixed in 4.0.3 and newer.

Please upgrade to the latest release and file a bug if you still run into this.

[1]: https://www.varnish-cache.org/trac/ticket/1539

On Fri, Jun 24, 2016 at 9:06 PM, Mahesh Ollalwar <mahesho at outlook.com> wrote:
> Can anyone help me on below error. looks like core dump is happening and
> child is getting restarted. This is happening recently without any change to
> the VCL and any system configuration.
>
> During this time the requests are getting terminated abruptly.
>
> Thanks in advance.
>
>
> Varnish version - varnish-4.0.2
>
>
> Jun 24 23:49:30 Prod-CACHE2 abrt[32341]: Saved core dump of pid 26526
> (/product/varnish/sbin/varnishd) to
> /var/spool/abrt/ccpp-2016-06-24-23:48:19-26526 (9034817536 bytes)
> Jun 24 23:49:30 Prod-CACHE2 abrtd: Directory
> 'ccpp-2016-06-24-23:48:19-26526' creation detected
> Jun 24 23:49:31 Prod-CACHE2 abrtd: Executable
> '/product/varnish/sbin/varnishd' doesn't belong to any package and
> ProcessUnpackaged is set to 'no'
> Jun 24 23:49:31 Prod-CACHE2 abrtd: 'post-create' on
> '/var/spool/abrt/ccpp-2016-06-24-23:48:19-26526' exited with 1
> Jun 24 23:49:31 Prod-CACHE2 abrtd: Deleting problem directory
> '/var/spool/abrt/ccpp-2016-06-24-23:48:19-26526'
> Jun 24 23:49:31 Prod-CACHE2 varnishd[26524]: Child (26526) not responding to
> CLI, killing it.
> Jun 24 23:49:31 Prod-CACHE2 varnishd[26524]: Child (26526) not responding to
> CLI, killing it.
> Jun 24 23:49:31 Prod-CACHE2 varnishd[26524]: Child (26526) died signal=6
> (core dumped)
> Jun 24 23:49:31 Prod-CACHE2 varnishd[26524]: Child (26526) Panic
> message:#012Assert error in cnt_lookup(), cache/cache_req_fsm.c line
> 430:#012  Condition((oc->exp_flags & (1<<7)) == 0) not true.#012thread =
> (cache-worker)#012ident =
> Linux,2.6.32-431.11.2.el6.x86_64,x86_64,-smalloc,-smalloc,-hcritbit,epoll#012Backtrace:#012
> 0x4310e5: pan_ic+0xc5#012  0x434c30: cnt_purge+0x0#012  0x436461:
> CNT_Request+0x241#012  0x42b73b: HTTP1_Session+0x53b#012  0x439f98:
> ses_req_pool_task+0x68#012  0x43a6d4: SES_pool_accept_task+0x2a4#012
> 0x433b87: Pool_Work_Thread+0x357#012  0x4467d9: wrk_thread_real+0xb9#012
> 0x3c25e079d1: /lib64/libpthread.so<http://libpthread.so>.0() [0x3c25e079d1]#012  0x3c25ae8b6d:
> /lib64/libc.so<http://libc.so>.6(clone+0x6d) [0x3c25ae8b6d]#012req = 0x7f83baaf2cb0 {#012
> sp = 0x7f84dd71c430, vxid = 1108394165,  step = R_STP_LOOKUP,#012  req_body
> = R_BODY_NONE,#012  restarts = 0, esi_level = 0#012  sp = 0x7f84dd71c430
> {#012    fd = 64, vxid = 34652201,#012    client = 10.254.9.78<http://10.254.9.78> 61217,#012
> step = S_STP_WORKING,#012  },#012  worker = 0x7f8574c0cc40 {#012    ws =
> 0x7f8574c0ce58 {#012      id = "wrk",#012      {s,f,r,e} =
> {0x7f8574c0ac30,0x7f8574c0ac30,(nil),+8192},#012    },#012  VCL::method =
> 0x0,#012  VCL::return = deliver,#012  },#012  ws = 0x7f83baaf2e48 {#012
> id = "req",#012    {s,f,r,e} = {0x7f83baaf4ca0,+1320,(nil),+57360},#012
> },#012  http[req] = {#012    ws = 0x7f83baaf2e48[req]#012      "GET",#012
> "/search-engine-web/SearchContents?action=SearchContents&moreFilters=series%3A3197&appVersion=5.2.2&query=*&startIndex=0&searchOrder=counter_week%20desc&maxResult=8&channel=ANDROID&type=CLIPS",#012
> "HTTP/1.1",#012      "host: xxx.xxx.com<http://xxx.xxx.com>",#012      "Accept-Encoding:
> gzip",#012      "Akamai-Origin-Hop: 4",#012      "True-Client-IP:
> 49.15.204.238<http://49.15.204.238>",#012      "User-Agent: Dalvik/2.1.0 (Linux; U; Android 5.1;
> 1201 Build/LMY47I)",#012      "Via: 1.1 v1-akamaitech.net<http://v1-akamaitech.net>(ghost)
> (AkamaiGHost), 1.1 v1-akamaitech.net<http://v1-akamaitech.net>(ghost) (AkamaiGHost), 1.1
> v1-akamaitech.net<http://v1-akamaitech.net>(ghost) (AkamaiGHost), 1.1 akamai.net<http://akamai.net>(ghost)
> (AkamaiGHost)",#012      "X-Aka
> Jun 24 23:49:31 Prod-SEARCH-CACHE2 varnishd[26524]: child (1277) Started
> Jun 24 23:49:31 Prod-SEARCH-CACHE2 varnishd[26524]: Child (1277) said Child
> starts
>
>
> Thanks,
>
> Mahesh.
>
>
> _______________________________________________
> varnish-misc mailing list
> varnish-misc at varnish-cache.org
> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc



--
Dag Haavi Finstad
Software Developer | Varnish Software
Mobile: +47 476 64 134
We Make Websites Fly!
_______________________________________________
varnish-misc mailing list
varnish-misc at varnish-cache.org
https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc<//prolinks.rediffmailpro.com/cgi-bin/prored.cgi?red=https%3A%2F%2Fwww%2Evarnish%2Dcache%2Eorg%2Flists%2Fmailman%2Flistinfo%2Fvarnish%2Dmisc&rediffng=0>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/varnish-misc/attachments/20160914/1babd81d/attachment.html>


More information about the varnish-misc mailing list