[Varnish] #1842: Assert error in hsh_rush()

[Varnish]

#1842: Assert error in hsh_rush()
----------------------+----------------------
 Reporter:  Sesse     |       Owner:
     Type:  defect    |      Status:  new
 Priority:  normal    |   Milestone:
Component:  varnishd  |     Version:  unknown
 Severity:  normal    |  Resolution:
 Keywords:            |
----------------------+----------------------
Changes (by lkarsten):

 * component:  build => varnishd


Old description:

> Hi,
>
> I ran 4.1.1-beta1 on Debian jessie (from your packages). Immediately as
> the load increased, it started crashing repeatedly. I picked out the last
> panic, and got this:
>
> Last panic at: Sat, 16 Jan 2016 13:39:46 GMT
> "Assert error in hsh_rush(), cache/cache_hash.c line 526:
>   Condition((wl) != NULL) not true.
> errno = 11 (Resource temporarily unavailable)
> thread = (cache-worker)
> version = varnish-4.1.1-beta1 revision 1fd0d39
> ident = Linux,4.3.0,x86_64,-junix,-smalloc,-smalloc,-hcritbit,epoll
> Backtrace:
>   0x433985: varnishd() [0x433985]
>   0x427e5a: varnishd() [0x427e5a]
>   0x428fc1: varnishd(HSH_DerefObjHead+0xa1) [0x428fc1]
>   0x4299c5: varnishd(HSH_Lookup+0x825) [0x4299c5]
>   0x436989: varnishd(CNT_Request+0x5b9) [0x436989]
>   0x44e713: varnishd(HTTP1_Session+0xe3) [0x44e713]
>   0x439d4d: varnishd(SES_Proto_Req+0x5d) [0x439d4d]
>   0x449062: varnishd() [0x449062]
>   0x4494db: varnishd() [0x4494db]
>   0x7f1c43b940a4: libpthread.so.0(+0x80a4) [0x7f1c43b940a4]
> req = 0x7f1c2f00f020 {
>   vxid = 229379, step = R_STP_LOOKUP,
>   req_body = R_BODY_NONE,
>   restarts = 0, esi_level = 0,
>   sp = 0x7f1c30c0e420 {
>     fd = 35, vxid = 229377,
>     client = 84.211.184.160 49650,
>     step = S_STP_H1PROC,
>   },
>   worker = 0x7f1c434c5c80 {
>     stack = {0x7f1c434c6000 -> 0x7f1c434ba000},
>     ws = 0x7f1c434c5e78 {
>       id = \"wrk\",
>       {s,f,r,e} = {0x7f1c434c5420,0x7f1c434c5420,(nil),+2040},
>     },
>     VCL::method = HASH,
>     VCL::return = lookup,
>     VCL::methods = {},
>   },
>   ws = 0x7f1c2f00f200 {
>     id = \"req\",
>     {s,f,r,e} = {0x7f1c2f011000,+608,+90104,+90104},
>   },
>   http_conn = 0x7f1c2f00f128 {
>     fd = 35,
>     doclose = NULL,
>     ws = 0x7f1c2f00f200,
>     {rxbuf_b, rxbuf_e} = {0x7f1c2f011000, 0x7f1c2f0111a3},
>     {pipeline_b, pipeline_e} = {(nil), (nil)},
>     content_length = -1,
>     body_status = none,
>     first_byte_timeout = 0.000000,
>     between_bytes_timeout = 0.000000,
>   },
>   http[req] = 0x7f1c2f00f298 {
>     ws[req] = 0x7f1c2f00f200,
>     hdrs {
>       \"GET\",
>       \"/analysis.pl?ims=1452951576000&unique=0.15300315176136792\",
>       \"HTTP/1.1\",
>       \"Host: analysis.sesse.net\",
>       \"Connection: keep-alive\",
>       \"Accept: */*\",
>       \"X-Requested-With: XMLHttpRequest\",
>       \"User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64)
> AppleWebKit/537.36 (KHTML, like Gecko) Chrome/47.0.2526.106
> Safari/537.36\",
>       \"Referer: http://analysis.sesse.net/\",
>       \"Accept-Language: en-US,en;q=0.8,nb;q=0.6\",
>       \"DNT: 1\",
>       \"X-Forwarded-For: 84.211.184.160, 84.211.184.160\",
>       \"x-analysis-backend: backend1\",
>       \"Accept-Encoding: gzip\",
>     },
>   },
>   vcl = {
>     temp = warm
>     srcname = {
>       \"/etc/varnish/default.vcl\",
>       \"Builtin\",
>     },
>   },
>   flags = {
>   },
> },
>
> "

New description:

 Hi,

 I ran 4.1.1-beta1 on Debian jessie (from your packages). Immediately as
 the load increased, it started crashing repeatedly. I picked out the last
 panic, and got this:

 {{{
 Last panic at: Sat, 16 Jan 2016 13:39:46 GMT
 "Assert error in hsh_rush(), cache/cache_hash.c line 526:
   Condition((wl) != NULL) not true.
 errno = 11 (Resource temporarily unavailable)
 thread = (cache-worker)
 version = varnish-4.1.1-beta1 revision 1fd0d39
 ident = Linux,4.3.0,x86_64,-junix,-smalloc,-smalloc,-hcritbit,epoll
 Backtrace:
   0x433985: varnishd() [0x433985]
   0x427e5a: varnishd() [0x427e5a]
   0x428fc1: varnishd(HSH_DerefObjHead+0xa1) [0x428fc1]
   0x4299c5: varnishd(HSH_Lookup+0x825) [0x4299c5]
   0x436989: varnishd(CNT_Request+0x5b9) [0x436989]
   0x44e713: varnishd(HTTP1_Session+0xe3) [0x44e713]
   0x439d4d: varnishd(SES_Proto_Req+0x5d) [0x439d4d]
   0x449062: varnishd() [0x449062]
   0x4494db: varnishd() [0x4494db]
   0x7f1c43b940a4: libpthread.so.0(+0x80a4) [0x7f1c43b940a4]
 req = 0x7f1c2f00f020 {
   vxid = 229379, step = R_STP_LOOKUP,
   req_body = R_BODY_NONE,
   restarts = 0, esi_level = 0,
   sp = 0x7f1c30c0e420 {
     fd = 35, vxid = 229377,
     client = 84.211.184.160 49650,
     step = S_STP_H1PROC,
   },
   worker = 0x7f1c434c5c80 {
     stack = {0x7f1c434c6000 -> 0x7f1c434ba000},
     ws = 0x7f1c434c5e78 {
       id = \"wrk\",
       {s,f,r,e} = {0x7f1c434c5420,0x7f1c434c5420,(nil),+2040},
     },
     VCL::method = HASH,
     VCL::return = lookup,
     VCL::methods = {},
   },
   ws = 0x7f1c2f00f200 {
     id = \"req\",
     {s,f,r,e} = {0x7f1c2f011000,+608,+90104,+90104},
   },
   http_conn = 0x7f1c2f00f128 {
     fd = 35,
     doclose = NULL,
     ws = 0x7f1c2f00f200,
     {rxbuf_b, rxbuf_e} = {0x7f1c2f011000, 0x7f1c2f0111a3},
     {pipeline_b, pipeline_e} = {(nil), (nil)},
     content_length = -1,
     body_status = none,
     first_byte_timeout = 0.000000,
     between_bytes_timeout = 0.000000,
   },
   http[req] = 0x7f1c2f00f298 {
     ws[req] = 0x7f1c2f00f200,
     hdrs {
       \"GET\",
       \"/analysis.pl?ims=1452951576000&unique=0.15300315176136792\",
       \"HTTP/1.1\",
       \"Host: analysis.sesse.net\",
       \"Connection: keep-alive\",
       \"Accept: */*\",
       \"X-Requested-With: XMLHttpRequest\",
       \"User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36
 (KHTML, like Gecko) Chrome/47.0.2526.106 Safari/537.36\",
       \"Referer: http://analysis.sesse.net/\",
       \"Accept-Language: en-US,en;q=0.8,nb;q=0.6\",
       \"DNT: 1\",
       \"X-Forwarded-For: 84.211.184.160, 84.211.184.160\",
       \"x-analysis-backend: backend1\",
       \"Accept-Encoding: gzip\",
     },
   },
   vcl = {
     temp = warm
     srcname = {
       \"/etc/varnish/default.vcl\",
       \"Builtin\",
     },
   },
   flags = {
   },
 },
 }}}

--

-- 
Ticket URL: <https://www.varnish-cache.org/trac/ticket/1842#comment:1>
Varnish <https://varnish-cache.org/>
The Varnish HTTP Accelerator