Varnish 5.2.0 child panic
Hugues Alary
hugues at betabrand.com
Tue Nov 28 19:48:15 UTC 2017
Hi there!
In the last 18 days, varnish reports its child died 8 times.
We had more traffic than usual those past few days, but I don't think it
was anything Varnish can't handle.
We run Varnish on Kubernetes on GCP.
I was able to retrieve the last panic via varnishadm (see below). The other
panics are the same as the one below (same assertion failing, same line
number). (I would send them too, but it's amazingly difficult to get logs
out of google cloud).
If the panic is not enough, I can send our configuration file on request.
Panic at: Tue, 28 Nov 2017 13:44:30 GMT
Assert error in HSH_Lookup(), cache/cache_hash.c line 432:
Condition((vary) != 0) not true.
version = varnish-5.2.0 revision 4c4875cbf, vrt api = 6.1
ident = Linux,4.4.64+,x86_64,-junix,-smalloc,-smalloc,-hcritbit,epoll
now = 1755374.451854 (mono), 1511876668.180978 (real)
Backtrace:
0x556f4d169e36: varnishd(+0x4ae36) [0x556f4d169e36]
0x556f4d1b4b80: varnishd(VAS_Fail+0x40) [0x556f4d1b4b80]
0x556f4d15f1b2: varnishd(HSH_Lookup+0xcb2) [0x556f4d15f1b2]
0x556f4d16e14f: varnishd(CNT_Request+0xedf) [0x556f4d16e14f]
0x556f4d18dda2: varnishd(+0x6eda2) [0x556f4d18dda2]
0x556f4d18525c: varnishd(+0x6625c) [0x556f4d18525c]
0x556f4d185780: varnishd(+0x66780) [0x556f4d185780]
0x7f13aa27c494: /lib/x86_64-linux-gnu/libpthread.so.0(+0x7494)
[0x7f13aa27c494]
0x7f13a9fbeaff: /lib/x86_64-linux-gnu/libc.so.6(clone+0x3f)
[0x7f13a9fbeaff]
thread = (cache-worker)
thr.req = 0x7f0c615bf020 {
vxid = 59420066, transport = HTTP/1 {
state = HTTP1::Proc
}
step = R_STP_LOOKUP,
req_body = R_BODY_NONE,
restarts = 0, esi_level = 0,
sp = 0x7f12be06de20 {
fd = 82, vxid = 59420065,
t_open = 1511876667.587429,
t_idle = 1511876667.587429,
transport = HTTP/1 {
state = HTTP1::Proc
}
client = 10.44.11.6 50566,
privs = 0x7f12be06de88 {
},
},
worker = 0x7f1374734dd0 {
stack = {0x7f1374735000 -> 0x7f13746a2000},
ws = 0x7f1374734e78 {
id = \"wrk\",
{s, f, r, e} = {0x7f1374734190, +0, (nil), +2040},
},
VCL::method = DELIVER,
VCL::return = deliver,
VCL::methods = {},
},
ws = 0x7f0c615bf208 {
id = \"req\",
{s, f, r, e} = {0x7f0c615c1008, +3920, +516080, +516080},
},
http_conn = 0x7f0c615bf130 {
fd = 82 (@0x7f12be06de38),
doclose = NULL,
ws = 0x7f0c615bf208 {
[Already dumped, see above]
},
{rxbuf_b, rxbuf_e} = {0x7f0c615c1008, 0x7f0c615c1e27},
{pipeline_b, pipeline_e} = {(nil), (nil)},
content_length = -1,
body_status = none,
first_byte_timeout = 0.000000,
between_bytes_timeout = 0.000000,
},
http[req] = 0x7f0c615bf2a0 {
ws = 0x7f0c615bf208 {
[Already dumped, see above]
},
hdrs {
\"GET\",
\"/api/rest/reviews/product/6430\",
\"HTTP/1.1\",
\"Host: www.betabrand.com\",
\"Accept-Encoding: gzip\",
\"CF-IPCountry: US\",
\"CF-RAY: 3c4dc30d9aae55b2-ORD\",
\"CF-Visitor: {\"scheme\":\"https\"}\",
\"user-agent: Mozilla/5.0 (iPhone; CPU iPhone OS 11_0_3 like Mac OS
X) AppleWebKit/604.1.38 (KHTML, like Gecko) Version/11.0 Mobile/15A432
Safari/604.1\",
\"accept-language: en-us\",
\"referer:
https://www.betabrand.com/womens/pants/womens-black-boot-flare-dress-pant-yoga-pants\
",
\"cookie:
optimizelyPendingLogEvents=%5B%22n%3Dimage_impression%26u%3Doeu1511755011199r0.44405056492852124%26wxhr%3Dtrue%26time%3D1511876665.858%26f%3D8443224466%2C8247617066%2C8411440184%2C7780900332%2C8106662898%26g%3D%22%5D;
sailthru_content=d9d08cd819b052bb46ae11d6f4f23820f9f47e5805766810fea9aeef193727f0f94155821eef8e7bdb5607beb554d9f12fd3caaaf239d3b03d10e1dc4b5fe514;
NaN_hash=d1015888FXVD1484095712135476;
__utma=207700493.554833224.1511755020.1511755021.1511876545.2;
__utmb=207700493.5.10.1511876545; __utmc=207700493;
__utmz=207700493.1511876545.2.2.utmcsr=Google|utmgclid=EAIaIQobChMIwbet_7Lh1wIVk7fACh0pGAaaEAAYASAAEgLFrPD_BwE|utmccn=Brand_General|utmcmd=Retention|utmctr=(not%20provided);
_ga=GA1.2.554833224.1511755020;
_gac_UA-17580748-1=1.1511876545.EAIaIQobChMIwbet_7Lh1wIVk7fACh0pGAaaEAAYASAAEgLFrPD_BwE;
_gac_UA-17580748-8=1.1511876545.EAIaIQobChMIwbet_7Lh1wIVk7fACh0pGAaaEAAYASAAEgLFrPD_BwE;
_gid=GA1.2.1787521728.1511876545;
_hp2_id.777713614=%7B%22userId%22%3A%221402506040644741%22%2C%22pageviewId%22%3A%226182215729724641%22%2C%22sessionId%22%3A%224386307704701158%22%2C%22identity%22%3Anull%2C%22trackerVersion%22%3A%223.0%22%7D;
_hp2_ses_props.777713614=%7B%22r%22%3A%22https%3A%2F%2Fwww.google.com
%2F%22%2C%22us%22%3A%22Google%22%2C%22um%22%3A%22Retention%22%2C%22ua%22%3A%22Brand_General%22%2C%22ts%22%3A1511876545351%2C%22d%22%3A%
22www.betabrand.com%22%2C%22h%22%3A%22%2Fsale%22%7D;
_sp_id.ac93=0115679b-fdb3-417c-9994-b1387273d811.1511755021.2.1511876663.1511755037.2396b5db-5ad9-4e6e-9061-f19c13780fc6;
_sp_ses.ac93=*;
optimizelyBuckets=%7B%228443224466%22%3A%228449793114%22%2C%228247617066%22%3A%228249041800%22%2C%228411440184%22%3A%228407050657%22%2C%228106662898%22%3A%228109860945%22%7D;
_gat_UA-17580748-8=1; frontend=deadbeefdeadbeefdeadbeefaa;
__zlcmid=jhgvWpO31r9clR; __utmt_UA-17580748-1=1; _uetsid=_ueta633ebda;
_bbli=0; fs_uid=www.fullstory.com`1bTW`4950547299041280:5668600916475904;
uuid=740c5206-78d1-4902-8896-a5e8e90eb7c8; bb-seed=0.01224735359031548;
betabrand-campaign-first=Brand_General,Google,Retention,undefined,undefined,Tue
Nov 28 2017 07:42:23 GMT-0600 (CST);
betabrand-campaign-last=Brand_General,Google,Retention,undefined,undefined,Tue
Nov 28 2017 07:42:23 GMT-0600 (CST);
betabrand-campaign-session=Brand_General,Google,Retention,undefined,undefined,Tue
Nov 28 2017 07:42:23 GMT-0600 (CST);
optimizelyEndUserId=oeu1511755011199r0.44405056492852124;
optimizelySegments=%7B%22172841198%22%3A%22campaign%22%2C%22172864896%22%3A%22safari%22%2C%22173083061%22%3A%22true%22%2C%223972941021%22%3A%22brand_general%22%7D;
__qca=P0-1874833837-1511755021423;
sailthru_visitor=b78950af-8d6d-4cb6-8d54-a497f97dd6ba; betabrand-timing=1;
3060738.3440491=e5d3965e-bb79-4cde-b307-d532ffdec44c;
tracker_device=34b4b287-87e9-4418-bef5-28ab27d9038c; EcommTest1=2;
__cfduid=dd480648d530b3001597f1e574200f3611511755011\",
\"CF-Connecting-IP: 73.44.212.19\",
\"X-Forwarded-Proto: https\",
\"Connection: close\",
\"X-Request-Start: t=1511876667586\",
\"X-Queue-Start: t=1511876667586\",
\"X-Unique-ID: 0A80002E:8771_0A2C0B06:01BB_5A1D683B_F1ABD22:0009\",
\"X-Forwarded-For: 73.44.212.19, 10.128.0.46, 10.44.11.6,
10.44.11.6\",
\"X-PSA-Blocking-Rewrite: betabrand-pagespeed\",
\"Accept: application/json\",
},
},
vcl = {
name = \"boot\",
busy = 92,
discard = 0,
state = auto,
temp = warm,
conf = {
srcname = {
\"/etc/varnish/default.vcl\",
\"Builtin\",
},
},
},
vmods = {
std = {Varnish 5.2.0 4c4875cbf, 0.0},
directors = {Varnish 5.2.0 4c4875cbf, 0.0},
},
flags = {
},
},
thr.busyobj = (nil) {
},
Thanks for your help!
Cheers,
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/varnish-misc/attachments/20171128/fa9d1dae/attachment-0001.html>
More information about the varnish-misc
mailing list