Varnish Panic
Yiannis Karayiannidis
y.karayiannidis at stoiximan.gr
Thu Mar 9 22:15:27 CET 2017
Hi all,
We had a stragne panic problem with our varnish server
i attach some log lines
any help will be more than appriciated.
*Mar 9 20:13:03 lin-varnish03 varnishd[5295]: Child (5297) not responding
to CLI, killed it.*
*Mar 9 20:13:03 lin-varnish03 varnishd[5295]: Unexpected reply from ping:
400 CLI communication error (hdr)*
*Mar 9 20:13:04 lin-varnish03 varnishd[5295]: Child (5297) died signal=6*
*Mar 9 20:13:04 lin-varnish03 varnishd[5295]: Child (5297) Last panic at:
Thu, 09 Mar 2017 20:13:04 GMT#012"Assert error in mpl_alloc(),
cache/cache_mempool.c line 79:#012 Condition((mi) != 0) not true.#012errno
= 12 (Cannot allocate memory)#012thread = (cache-worker)#012version =
varnish-4.1.5 revision 2c82b1c#012ident =
Linux,3.10.0-514.6.1.el7.x86_64,x86_64,-junix,-smalloc,-smalloc,-hcritbit,epoll*#012now
= 1254528.822470 (mono), 1489090356.283966 (real)#012Backtrace:#012req =
0x7f6d185f6020 {#012 vxid = 1027769622, step = R_STP_PASS,#012 req_body =
R_BODY_NONE,#012 restarts = 0, esi_level = 0,#012 sp = 0x7f6b465dd220
{#012 fd = 46366, vxid = 1027769621,#012 client = 127.0.0.1
45267,#012 t_open = 1489090356.283755,#012 t_idle =
1489090356.283755,#012 step = S_STP_H1PROC,#012 },#012 worker =
0x7f6e12a0ac40 {#012 stack = {0x7f6e12a0b000 -> 0x7f6e129ff000},#012
ws = 0x7f6e12a0ae38 {#012 id = \"wrk\",#012 {s,f,r,e} =
{0x7f6e12a0a3e0,0x7f6e12a0a3e0,(nil),+2040},#012 },#012 VCL::method =
PASS,#012 VCL::return = fetch,#012 VCL::methods = {RECV, PASS,
HASH},#012 },#012 ws = 0x7f6d185f6200 {#012 id = \"req\",#012
{s,f,r,e} = {0x7f6d185f8000,+5024,(nil),+57336},#012 },#012 http_conn =
0x7f6d185f6128 {#012 fd = 46366,#012 doclose = NULL,#012 ws =
0x7f6d185f6200,#012 {rxbuf_b, rxbuf_e} = {0x7f6d185f8000,
0x7f6d185f8774},#012 {pipeline_b, pipeline_e} = {(nil), (nil)},#012
content_length = -1,#012 body_status = none,#012 first_byte_timeout
= 0.000000,#012 between_bytes_timeout = 0.000000,#012 },#012 http[req]
= 0x7f6d185f6298 {#012 ws[req] = 0x7f6d185f6200,#012 hdrs {#012
\"GET\",#012 \"/out?only=true&_=1489089262776\",#012
\"HTTP/1.1\",#012 \"Host: www.xxx.com\",#012
\"X-Forwarded-Proto: https\",#012 \"Connection: close\",#012
\"Accept-Encoding: gzip\",#012 \"CF-IPCountry: EN\",#012
\"CF-RAY: 33d0b0a4dae65cb7-ATH\",#012 \"CF-Visitor:
{\"scheme\":\"https\"}\",#012 \"Accept: application/json,
text/javascript, */*; q=0.01\",#012 \"X-If-None-Match:
RtbjjaE-8Jrgzu9Yp9qIPQ2\",#012 \"X-Requested-With:
XMLHttpRequest\",#012 \"User-Agent: Mozilla/5.0 (Windows NT 6.1;
WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/56.0.2924.87
Safari/537.36\",#012 \"Referer: https://www.xxx.com/ntlive/\",#012
\"Accept-Language: en-GR,en;q=0.8,hu;q=0.6,en;q=0.4\",#012
\"CF-Connecting-IP: 122.85.108.207\",#012 \"X-Forwarded-For:
122.85.108.207, 122.85.108.207, 127.0.0.1\",#012 \"sticky:
stx98.141\",#012 \"Cookie: sticky=stx89.746;
_ga=GA1.2.17060266.132423773; _tz=120;
uth1=F9A34D852B9715121425DB5B1393F63D30003234A8AC5811C9209E59;
AUTH=Yyg1I2QAZ7dsiHxSrs6cX4g9pxcOvLh2iPaF2NoEjcnoVa2p6fmNA5oyhfueuWMoMI1Kr3dR8Voi0er_Bm5nR6aNa3UQ251tg=\",#012
\"X-defHash: /out?liveonly=true&_=1489089262776 + www.xxx.com\",#012
},#012 },#012 vcl = {#012 busy = 47089#012 discard = 0,#012
state = auto,#012 temp = warm,#012 conf =
{#012 srcname = {#012 \"/etc/varnish/default.vcl\",#012
\"Builtin\",#012 \"/etc/varnish/origins2.vcl\",#012
\"/etc/varnish/origins1.com.vcl\",#012
\"/etc/varnish/origins.vcl\",#012
\"/etc/varnish/origins-stagwqwqi-trick.vcl\",#012
\"/etc/varnish/bwewanlist.vcl\",#012 },#012 },#012 },#012
objcore[REQ] = 0x7f6b2d7b3080 {#012 refcnt = 1,#012 flags =
0x10a,#012 exp_flags = 0x0,#012 exp = { 0.000000, 0.000000, 0.000000,
0.000000 }#012 objhead = 0x7f6e3f4f03e0,#012 stevedore = (nil),#012
},#012 flags = {#012 },#012},#012#012"
Mar 9 20:13:04 lin-varnish03 varnishd[5295]: Child cleanup complete
Mar 9 20:13:04 lin-varnish03 varnishd[5295]: Child (5892) Started
Mar 9 20:13:04 lin-varnish03 varnishd[5295]: Child (5892) said Child starts
*Mar 9 20:13:54 lin-varnish03 varnishd[5295]: Child (5892) not responding
to CLI, killed it.*
*Mar 9 20:13:54 lin-varnish03 varnishd[5295]: Unexpected reply from ping:
400 CLI communication error (hdr)*
*Mar 9 20:13:55 lin-varnish03 varnishd[5295]: Child (5892) died signal=6*
*Mar 9 20:13:55 lin-varnish03 varnishd[5295]: Child (5892) Last panic at:
Thu, 09 Mar 2017 20:13:55 GMT#012"Missing errorhandling code in
hsh_NewObjCore(), cache/cache_hash.c line 76:#012 Condition((oc) != 0) not
true.#012errno = 12 (Cannot allocate memory)#012thread =
(cache-worker)#012version = varnish-4.1.5 revision 2c82b1c#012ident =
Linux,3.10.0-514.6.1.el7.x86_64,x86_64,-junix,-smalloc,-smalloc,-hcritbit,epoll#*012now
= 1254528.822470 (mono), 1489090356.283966 (real)#012Backtrace:#012req =
0x7f6d185f6020 {#012 vxid = 1027769622, step = R_STP_PASS,#012 req_body =
R_BODY_NONE,#012 restarts = 0, esi_level = 0,#012 sp = 0x7f6b465dd220
{#012 fd = 46366, vxid = 1027769621,#012 client = 127.0.0.1
45267,#012 t_open = 1489090356.283755,#012 t_idle =
1489090356.283755,#012 step = S_STP_H1PROC,#012 },#012 worker =
0x7f6e12a0ac40 {#012 stack = {0x7f6e12a0b000 -> 0x7f6e129ff000},#012
ws = 0x7f6e12a0ae38 {#012 id = \"wrk\",#012 {s,f,r,e} =
{0x7f6e12a0a3e0,0x7f6e12a0a3e0,(nil),+2040},#012 },#012 VCL::method =
PASS,#012 VCL::return = fetch,#012 VCL::methods = {RECV, PASS,
HASH},#012 },#012 ws = 0x7f6d185f6200 {#012 id = \"req\",#012
{s,f,r,e} = {0x7f6d185f8000,+5024,(nil),+57336},#012 },#012 http_conn =
0x7f6d185f6128 {#012 fd = 46366,#012 doclose = NULL,#012 ws =
0x7f6d185f6200,#012 {rxbuf_b, rxbuf_e} = {0x7f6d185f8000,
0x7f6d185f8774},#012 {pipeline_b, pipeline_e} = {(nil), (nil)},#012
content_length = -1,#012 body_status = none,#012 first_byte_timeout
= 0.000000,#012 between_bytes_timeout = 0.000000,#012 },#012 http[req]
= 0x7f6d185f6298 {#012 ws[req] = 0x7f6d185f6200,#012 hdrs {#012
\"GET\",#012 \"/out?only=true&_=1489089262776\",#012
\"HTTP/1.1\",#012 \"Host: www.xxx.com\",#012
\"X-Forwarded-Proto: https\",#012 \"Connection: close\",#012
\"Accept-Encoding: gzip\",#012 \"CF-IPCountry: EN\",#012
\"CF-RAY: 33d0b0a4dae65cb7-ATH\",#012 \"CF-Visitor:
{\"scheme\":\"https\"}\",#012 \"Accept: application/json,
text/javascript, */*; q=0.01\",#012 \"X-If-None-Match:
RtbjjaE-8Jrgzu9Yp9qIPQ2\",#012 \"X-Requested-With:
XMLHttpRequest\",#012 \"User-Agent: Mozilla/5.0 (Windows NT 6.1;
WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/56.0.2924.87
Safari/537.36\",#012 \"Referer: https://www.xxx.com/ntlive/\",#012
\"Accept-Language: en-GR,en;q=0.8,hu;q=0.6,en;q=0.4\",#012
\"CF-Connecting-IP: 122.85.108.207\",#012 \"X-Forwarded-For:
122.85.108.207, 122.85.108.207, 127.0.0.1\",#012 \"sticky:
stx98.141\",#012 \"Cookie: sticky=stx89.746;
_ga=GA1.2.17060266.132423773; _tz=120;
uth1=D30A30ABC4B7D4AA7A0639A37F1C84B572B396E51335C456A710F72B1411990C4CF9A34D852B9715121425DB5B1393F63D30003234A8AC5811C9209E59;
AUTH=Yyg1I2QAZ7dsiHxSrs6cX4g9pxcOvLh2MI1Kr3dR8Voi0er_Bm5nR6aNa3UQ251tg=\",#012
\"X-defHash: /out?only=true&_=1489089262776 + www.xxx.com\",#012
},#012 },#012 vcl = {#012 busy = 47089#012 discard = 0,#012
state = auto,#012 temp = warm,#012 conf =
{#012 srcname = {#012 \"/etc/varnish/default.vcl\",#012
\"Builtin\",#012 \"/etc/varnish/origins2.vcl\",#012
\"/etc/varnish/origins1.com.vcl\",#012
\"/etc/varnish/origins.vcl\",#012
\"/etc/varnish/origins-stagwqwqi-trick.vcl\",#012
\"/etc/varnish/bwewanlist.vcl\",#012 },#012 },#012 },#012
objcore[REQ] = 0x7f6b2d7b3080 {#012 refcnt = 1,#012 flags =
0x10a,#012 exp_flags = 0x0,#012 exp = { 0.000000, 0.000000, 0.000000,
0.000000 }#012 objhead = 0x7f6e3f4f03e0,#012 stevedore = (nil),#012
},#012 flags = {#012 },#012},#012#012"
Mar 9 20:13:55 lin-varnish03 varnishd[5295]: Child cleanup complete
Mar 9 20:13:55 lin-varnish03 varnishd[5295]: Child (2878) Started
Mar 9 20:13:56 lin-varnish03 varnishd[5295]: Child (2878) said Child starts
Mar 9 20:13:56 lin-varnish03 varnishd[5295]: CLI telnet 127.0.0.1 35892
127.0.0.1 6082 Rd auth
9a804ba5663b2525c372565714aedbb2a11b899cd24bf3d7d33d4969c440a05c
Mar 9 20:13:56 lin-varnish03 varnishd[5295]: CLI telnet 127.0.0.1 35892
127.0.0.1 6082 Wr 200 -----------------------------#012Varnish Cache CLI
1.0#012-----------------------------#012Linux,3.10.0-514.6.1.el7.x86_64,x86_64,-junix,-smalloc,-smalloc,-hcritbit#012varnish-4.1.5
revision 2c82b1c#012#012Type 'help' for command list.#012Type 'quit' to
close CLI session.
Mar 9 20:14:36 lin-varnish03 varnishd[5295]: Child (2878) not responding
to CLI, killed it.
Mar 9 20:14:36 lin-varnish03 varnishd[5295]: Unexpected reply from ping:
400 CLI communication error (hdr)
Mar 9 20:14:36 lin-varnish03 varnishd[5295]: Child (2878) not responding
to CLI, killed it.
Mar 9 20:14:36 lin-varnish03 varnishd[5295]: Unexpected reply from ping:
400 CLI communication error
Mar 9 20:14:37 lin-varnish03 varnishd[5295]: Child (2878) died signal=11
Mar 9 20:14:37 lin-varnish03 varnishd[5295]: Child (2878) Last panic at:
Thu, 09 Mar 2017 20:14:37 GMT#012"Missing errorhandling code in
hsh_NewObjCore(), cache/cache_hash.c line 76:#012 Condition((oc) != 0) not
true.errno = 12 (Cannot allocate memory)#012thread =
(cache-worker)#012version = varnish-4.1.5 revision 2c82b1c#012ident =
Linux,3.10.0-514.6.1.el7.x86_64,x86_64,-junix,-smalloc,-smalloc,-hcritbit,epoll#012now
= 1254622.398113 (mono), 1489090449.859610 (real)#012"
Mar 9 20:14:37 lin-varnish03 varnishd[5295]: Child cleanup complete
Mar 9 20:14:37 lin-varnish03 varnishd[5295]: Child (35949) Started
Mar 9 20:14:37 lin-varnish03 varnishd[5295]: Child (35949) said Child
starts
Regards
Yianni
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/varnish-misc/attachments/20170309/fc980c88/attachment-0001.html>
More information about the varnish-misc
mailing list