varnish core dump
Mahesh Ollalwar
mahesho at outlook.com
Mon Sep 12 11:07:29 CEST 2016
Hi,
Thanks for the response.
We have upgraded to latest stable version 4.1.3 but still see panics. below are the logs.
Sep 12 12:45:18 Prod-SHARED-CACHE6 abrt[32688]: Saved core dump of pid 24551 (/product/varnish/sbin/varnishd) to /var/spool/abrt/ccpp-2016-09-12-12:45:15-24551 (585650176 bytes)
Sep 12 12:45:18 Prod-SHARED-CACHE6 abrtd: Directory 'ccpp-2016-09-12-12:45:15-24551' creation detected
Sep 12 12:45:19 Prod-SHARED-CACHE6 abrtd: Executable '/product/varnish/sbin/varnishd' doesn't belong to any package and ProcessUnpackaged is set to 'no'
Sep 12 12:45:19 Prod-SHARED-CACHE6 abrtd: 'post-create' on '/var/spool/abrt/ccpp-2016-09-12-12:45:15-24551' exited with 1
Sep 12 12:45:19 Prod-SHARED-CACHE6 abrtd: Deleting problem directory '/var/spool/abrt/ccpp-2016-09-12-12:45:15-24551'
Sep 12 12:45:19 Prod-SHARED-CACHE6 varnishd[11105]: Child (24551) not responding to CLI, killing it.
Sep 12 12:45:20 Prod-SHARED-CACHE6 varnishd[11105]: Child (24551) died signal=6 (core dumped)
Sep 12 12:45:20 Prod-SHARED-CACHE6 varnishd[11105]: Child (24551) Last panic at: Mon, 12 Sep 2016 07:15:20 GMT#012"Assert error in http_EstimateWS(), cache/cache_http.c line 793:#012 Condition((fm->hd[u].b) != 0) not true.#012thread = (cache-worker)#012version = varnish-4.1.3 revision 5e3b6d2#012ident = Linux,2.6.32-431.11.2.el6.x86_64,x86_64,-junix,-smalloc,-smalloc,-hcritbit,epoll#012Backtrace:#012 0x432f13: pan_ic+0x163#012 0x42dec5: http_EstimateWS+0x435#012 0x421874: vbf_beresp2obj+0x84#012 0x422cb5: vbf_stp_error+0x235#012 0x42366b: vbf_fetch_thread+0x1db#012 0x449825: WRK_Thread+0x4a5#012 0x449c5b: pool_thread+0x2b#012 0x3c25e079d1: libpthread.so.0() [0x3c25e079d1]#012 0x3c25ae8b6d: libc.so.6(clone+0x6d) [0x3c25ae8b6d]#012busyobj = 0x7f7f505208a0 {#012 ws = 0x7f7f50520960 {#012 OVERFLOWED id = \"Bo\",#012 {s,f,r,e} = {0x7f7f50522818,+57440,(nil),+57440},#012 },#012 refcnt = 2,#012 retries = 0, failed = 1, state = 1,#012 flags = {do_gzip, do_stream, is_gunzip},#012 director_req = 0x1a16130 {#012 vcl_name = farm_anon,#012 type = round-robin {#012 },#012 },#012 director_resp = 0x1a08e98 {#012 vcl_name = PROD_LB_BE_ANON_1,#012 type = backend {#012 display_name = boot.PROD_LB_BE_ANON_1,#012 ipv4 = 10.254.4.254,#012 port = 8080,#012 hosthdr = 10.254.4.254,#012 health=healthy, admin_health=probe, changed=1473664247.5,#012 n_conn = 2,#012 },#012 },#012 http[bereq] = 0x7f7f50520f28 {#012 ws[Bo] = 0x7f7f50520960,#012 hdrs {#012 \"GET\",#012 \"/AVS/besc?action=GetArrayContentList&channel=ANDROID&categoryId=7022\",#012 \"HTTP/1.1\",#012 \"host: www.example.com<http://www.example.com> \",#012 \"Accept-Encoding: gzip\",#012 \"Akamai-Origin-Hop: 4\",#012 \"True-Client-IP: 183.83.249.187\",#012 \"User-Agent: Dalvik/2.1.0 (Linux; U; Android 6.0.1; Le X526 Build/IIXOSOP5801607291S)\",#012 \"Via: 1.1 v1-akamaitech.net(ghost) (AkamaiGHost), 1.1 v1-akamaitech.net(ghost) (AkamaiGHost), 1.1 v1-akamaitech.net(ghost) (AkamaiGHost), 1.1 aka
Sep 12 12:45:20 Prod-SHARED-CACHE6 varnishd[11105]: Child (386) said Child starts
Thanks in advance,
Mahesh.
________________________________
From: Dag Haavi Finstad <daghf at varnish-software.com>
Sent: Monday, June 27, 2016 2:57 PM
To: Mahesh Ollalwar
Cc: varnish-misc at varnish-cache.org
Subject: Re: varnish core dump
Hi
I believe this might be #1539 [1], which is fixed in 4.0.3 and newer.
Please upgrade to the latest release and file a bug if you still run into this.
[1]: https://www.varnish-cache.org/trac/ticket/1539
On Fri, Jun 24, 2016 at 9:06 PM, Mahesh Ollalwar <mahesho at outlook.com> wrote:
> Can anyone help me on below error. looks like core dump is happening and
> child is getting restarted. This is happening recently without any change to
> the VCL and any system configuration.
>
> During this time the requests are getting terminated abruptly.
>
> Thanks in advance.
>
>
> Varnish version - varnish-4.0.2
>
>
> Jun 24 23:49:30 Prod-CACHE2 abrt[32341]: Saved core dump of pid 26526
> (/product/varnish/sbin/varnishd) to
> /var/spool/abrt/ccpp-2016-06-24-23:48:19-26526 (9034817536 bytes)
> Jun 24 23:49:30 Prod-CACHE2 abrtd: Directory
> 'ccpp-2016-06-24-23:48:19-26526' creation detected
> Jun 24 23:49:31 Prod-CACHE2 abrtd: Executable
> '/product/varnish/sbin/varnishd' doesn't belong to any package and
> ProcessUnpackaged is set to 'no'
> Jun 24 23:49:31 Prod-CACHE2 abrtd: 'post-create' on
> '/var/spool/abrt/ccpp-2016-06-24-23:48:19-26526' exited with 1
> Jun 24 23:49:31 Prod-CACHE2 abrtd: Deleting problem directory
> '/var/spool/abrt/ccpp-2016-06-24-23:48:19-26526'
> Jun 24 23:49:31 Prod-CACHE2 varnishd[26524]: Child (26526) not responding to
> CLI, killing it.
> Jun 24 23:49:31 Prod-CACHE2 varnishd[26524]: Child (26526) not responding to
> CLI, killing it.
> Jun 24 23:49:31 Prod-CACHE2 varnishd[26524]: Child (26526) died signal=6
> (core dumped)
> Jun 24 23:49:31 Prod-CACHE2 varnishd[26524]: Child (26526) Panic
> message:#012Assert error in cnt_lookup(), cache/cache_req_fsm.c line
> 430:#012 Condition((oc->exp_flags & (1<<7)) == 0) not true.#012thread =
> (cache-worker)#012ident =
> Linux,2.6.32-431.11.2.el6.x86_64,x86_64,-smalloc,-smalloc,-hcritbit,epoll#012Backtrace:#012
> 0x4310e5: pan_ic+0xc5#012 0x434c30: cnt_purge+0x0#012 0x436461:
> CNT_Request+0x241#012 0x42b73b: HTTP1_Session+0x53b#012 0x439f98:
> ses_req_pool_task+0x68#012 0x43a6d4: SES_pool_accept_task+0x2a4#012
> 0x433b87: Pool_Work_Thread+0x357#012 0x4467d9: wrk_thread_real+0xb9#012
> 0x3c25e079d1: /lib64/libpthread.so.0() [0x3c25e079d1]#012 0x3c25ae8b6d:
> /lib64/libc.so.6(clone+0x6d) [0x3c25ae8b6d]#012req = 0x7f83baaf2cb0 {#012
> sp = 0x7f84dd71c430, vxid = 1108394165, step = R_STP_LOOKUP,#012 req_body
> = R_BODY_NONE,#012 restarts = 0, esi_level = 0#012 sp = 0x7f84dd71c430
> {#012 fd = 64, vxid = 34652201,#012 client = 10.254.9.78 61217,#012
> step = S_STP_WORKING,#012 },#012 worker = 0x7f8574c0cc40 {#012 ws =
> 0x7f8574c0ce58 {#012 id = "wrk",#012 {s,f,r,e} =
> {0x7f8574c0ac30,0x7f8574c0ac30,(nil),+8192},#012 },#012 VCL::method =
> 0x0,#012 VCL::return = deliver,#012 },#012 ws = 0x7f83baaf2e48 {#012
> id = "req",#012 {s,f,r,e} = {0x7f83baaf4ca0,+1320,(nil),+57360},#012
> },#012 http[req] = {#012 ws = 0x7f83baaf2e48[req]#012 "GET",#012
> "/search-engine-web/SearchContents?action=SearchContents&moreFilters=series%3A3197&appVersion=5.2.2&query=*&startIndex=0&searchOrder=counter_week%20desc&maxResult=8&channel=ANDROID&type=CLIPS",#012
> "HTTP/1.1",#012 "host: xxx.xxx.com",#012 "Accept-Encoding:
> gzip",#012 "Akamai-Origin-Hop: 4",#012 "True-Client-IP:
> 49.15.204.238",#012 "User-Agent: Dalvik/2.1.0 (Linux; U; Android 5.1;
> 1201 Build/LMY47I)",#012 "Via: 1.1 v1-akamaitech.net(ghost)
> (AkamaiGHost), 1.1 v1-akamaitech.net(ghost) (AkamaiGHost), 1.1
> v1-akamaitech.net(ghost) (AkamaiGHost), 1.1 akamai.net(ghost)
> (AkamaiGHost)",#012 "X-Aka
> Jun 24 23:49:31 Prod-SEARCH-CACHE2 varnishd[26524]: child (1277) Started
> Jun 24 23:49:31 Prod-SEARCH-CACHE2 varnishd[26524]: Child (1277) said Child
> starts
>
>
> Thanks,
>
> Mahesh.
>
>
> _______________________________________________
> varnish-misc mailing list
> varnish-misc at varnish-cache.org
> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
--
Dag Haavi Finstad
Software Developer | Varnish Software
Mobile: +47 476 64 134
We Make Websites Fly!
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/varnish-misc/attachments/20160912/9c2d30d3/attachment-0001.html>
More information about the varnish-misc
mailing list