[Varnish] #508: Varnish 2.0.4 started crashing more often under load. Assert error in VRT_r_req_backend_healthy().

[Varnish]

#508: Varnish 2.0.4 started crashing more often under load. Assert error in
VRT_r_req_backend_healthy().
----------------------+-----------------------------------------------------
 Reporter:  nicholas  |       Owner:  phk
     Type:  defect    |      Status:  new
 Priority:  normal    |   Milestone:     
Component:  varnishd  |     Version:  2.0
 Severity:  normal    |    Keywords:     
----------------------+-----------------------------------------------------
 Varnish 2.0.4 started crashing more often under load. Load is ~1k hits/s.

 Red Hat Enterprise Linux Server release 5 (Tikanga)
 # uname -s -r -m -p -i -o
 Linux 2.6.18-92.1.13.el5 x86_64 x86_64 x86_64 GNU/Linux
 # rpm -qa | grep varnish
 varnish-libs-devel-2.0.4-1.el5
 varnish-2.0.4-1.el5
 varnish-debuginfo-2.0.4-1.el5
 varnish-libs-2.0.4-1.el5

 custom vcl, with heavy use of backends, health probes and directors. I can
 send vcl in private if necessary.

 /var/log/messages:
 May  8 06:45:35 rabbla varnishd[11699]: Child (24950) died signal=6 (core
 dumped)
 May  8 06:45:35 rabbla varnishd[11699]: Child (24950) Panic message:
 Assert error in VRT_r_req_backend_healthy(), cache_vrt.c line 663:
 Condition((sp->director) != NULL) not true.  thread = (cache-worker)sp =
 0x2aabeb9e0008 {   fd = 49, id = 49, xid = 478829260,   client =
 87.238.55.90:57525,   step = STP_DELIVER,   handling = error,   ws =
 0x2aabeb9e0078 {      id = "sess",     {s,f,r,e} =
 {0x2aabeb9e0808,,+6011,(nil),+16384},   },     worker = 0x5d742bd0 {
 },     vcl = {       srcname = {         "input",         "Default",
 "/etc/varnish/publications-begin.txt",         "/etc/varnish/include-
 clusters.
 vcl",         "/etc/varnish/include-clusters.vcl",         "/etc/varnish
 /include-clusters.vcl",         "/etc/varnish/include-clusters.vcl",
 "/etc/varnish/include-clusters.vcl",         "/etc/varnish/
 include-clusters.vcl",         "/etc/varnish/publications-end.txt",
 "/etc/varnish/include-clusters.vcl",         "/etc/varnish/include-
 clusters.vcl",         "/etc/varnish/include-clus

 (gdb) bt
 #0  0x0000003abde30155 in raise () from /lib64/libc.so.6
 #1  0x0000003abde31bf0 in abort () from /lib64/libc.so.6
 #2  0x000000000041af57 in pan_ic (func=<value optimized out>, file=<value
 optimized out>, line=<value optimized out>, cond=<value optimized out>,
 err=64, xxx=<value optimized out>) at cache_panic.c:317
 #3  0x00000000004200c5 in VRT_r_req_backend_healthy (sp=0x2aabebe4f008) at
 cache_vrt.c:663
 #4  0x00002aabeb5071c8 in ?? ()
 #5  0x000000005cc96bd0 in ?? ()
 #6  0x00002aabebe4f008 in ?? ()
 #7  0x00002aabebe4f008 in ?? ()
 #8  0x000000000041f416 in VCL_deliver_method (sp=0x2aabebe4f008) at
 ../../include /vcl_returns.h:59
 #9  0x000000000040f77a in cnt_deliver (sp=0x2aabebe4f008) at
 cache_center.c:166
 #10 0x00000000004104ba in CNT_Session (sp=0x5cc94b20) at steps.h:42
 #11 0x000000000041c4f2 in wrk_do_cnt_sess (w=0x5cc96bd0, priv=<value
 optimized out>) at cache_pool.c:398
 #12 0x000000000041bbbf in wrk_thread (priv=0x2ad8203331f0) at
 cache_pool.c:310
 #13 0x0000003abe6062f7 in start_thread () from /lib64/libpthread.so.0
 #14 0x0000003abded1b6d in clone () from /lib64/libc.so.6

-- 
Ticket URL: <http://varnish.projects.linpro.no/ticket/508>
Varnish <http://varnish.projects.linpro.no/>
The Varnish HTTP Accelerator