[Varnish] #508: Varnish 2.0.4 started crashing more often under load. Assert error in VRT_r_req_backend_healthy().
Varnish
varnish-bugs at projects.linpro.no
Sat May 9 12:01:05 CEST 2009
#508: Varnish 2.0.4 started crashing more often under load. Assert error in
VRT_r_req_backend_healthy().
----------------------+-----------------------------------------------------
Reporter: nicholas | Owner: phk
Type: defect | Status: new
Priority: normal | Milestone:
Component: varnishd | Version: 2.0
Severity: normal | Resolution:
Keywords: |
----------------------+-----------------------------------------------------
Old description:
> Varnish 2.0.4 started crashing more often under load. Load is ~1k hits/s.
>
> Red Hat Enterprise Linux Server release 5 (Tikanga)
> # uname -s -r -m -p -i -o
> Linux 2.6.18-92.1.13.el5 x86_64 x86_64 x86_64 GNU/Linux
> # rpm -qa | grep varnish
> varnish-libs-devel-2.0.4-1.el5
> varnish-2.0.4-1.el5
> varnish-debuginfo-2.0.4-1.el5
> varnish-libs-2.0.4-1.el5
>
> custom vcl, with heavy use of backends, health probes and directors. I
> can send vcl in private if necessary.
>
> /var/log/messages:
> May 8 06:45:35 rabbla varnishd[11699]: Child (24950) died signal=6 (core
> dumped)
> May 8 06:45:35 rabbla varnishd[11699]: Child (24950) Panic message:
> Assert error in VRT_r_req_backend_healthy(), cache_vrt.c line 663:
> Condition((sp->director) != NULL) not true. thread = (cache-worker)sp =
> 0x2aabeb9e0008 { fd = 49, id = 49, xid = 478829260, client =
> 87.238.55.90:57525, step = STP_DELIVER, handling = error, ws =
> 0x2aabeb9e0078 { id = "sess", {s,f,r,e} =
> {0x2aabeb9e0808,,+6011,(nil),+16384}, }, worker = 0x5d742bd0 {
> }, vcl = { srcname = { "input", "Default",
> "/etc/varnish/publications-begin.txt", "/etc/varnish/include-
> clusters.
> vcl", "/etc/varnish/include-clusters.vcl", "/etc/varnish
> /include-clusters.vcl", "/etc/varnish/include-clusters.vcl",
> "/etc/varnish/include-clusters.vcl", "/etc/varnish/
> include-clusters.vcl", "/etc/varnish/publications-end.txt",
> "/etc/varnish/include-clusters.vcl", "/etc/varnish/include-
> clusters.vcl", "/etc/varnish/include-clus
>
> (gdb) bt
> #0 0x0000003abde30155 in raise () from /lib64/libc.so.6
> #1 0x0000003abde31bf0 in abort () from /lib64/libc.so.6
> #2 0x000000000041af57 in pan_ic (func=<value optimized out>, file=<value
> optimized out>, line=<value optimized out>, cond=<value optimized out>,
> err=64, xxx=<value optimized out>) at cache_panic.c:317
> #3 0x00000000004200c5 in VRT_r_req_backend_healthy (sp=0x2aabebe4f008)
> at cache_vrt.c:663
> #4 0x00002aabeb5071c8 in ?? ()
> #5 0x000000005cc96bd0 in ?? ()
> #6 0x00002aabebe4f008 in ?? ()
> #7 0x00002aabebe4f008 in ?? ()
> #8 0x000000000041f416 in VCL_deliver_method (sp=0x2aabebe4f008) at
> ../../include /vcl_returns.h:59
> #9 0x000000000040f77a in cnt_deliver (sp=0x2aabebe4f008) at
> cache_center.c:166
> #10 0x00000000004104ba in CNT_Session (sp=0x5cc94b20) at steps.h:42
> #11 0x000000000041c4f2 in wrk_do_cnt_sess (w=0x5cc96bd0, priv=<value
> optimized out>) at cache_pool.c:398
> #12 0x000000000041bbbf in wrk_thread (priv=0x2ad8203331f0) at
> cache_pool.c:310
> #13 0x0000003abe6062f7 in start_thread () from /lib64/libpthread.so.0
> #14 0x0000003abded1b6d in clone () from /lib64/libc.so.6
New description:
Varnish 2.0.4 started crashing more often under load. Load is ~1k hits/s.
{{{
Red Hat Enterprise Linux Server release 5 (Tikanga)
# uname -s -r -m -p -i -o
Linux 2.6.18-92.1.13.el5 x86_64 x86_64 x86_64 GNU/Linux
# rpm -qa | grep varnish
varnish-libs-devel-2.0.4-1.el5
varnish-2.0.4-1.el5
varnish-debuginfo-2.0.4-1.el5
varnish-libs-2.0.4-1.el5
custom vcl, with heavy use of backends, health probes and directors. I can
send vcl in private if necessary.
/var/log/messages:
May 8 06:45:35 rabbla varnishd[11699]: Child (24950) died signal=6 (core
dumped)
May 8 06:45:35 rabbla varnishd[11699]: Child (24950) Panic message:
Assert error in VRT_r_req_backend_healthy(), cache_vrt.c line 663:
Condition((sp->director) != NULL) not true.
thread = (cache-worker)sp = 0x2aabeb9e0008 {
fd = 49, id = 49, xid = 478829260,
client = 87.238.55.90:57525,
step = STP_DELIVER,
handling = error,
ws = 0x2aabeb9e0078 {
id = "sess",
{s,f,r,e} = {0x2aabeb9e0808,,+6011,(nil),+16384},
},
worker = 0x5d742bd0 {
},
vcl = {
srcname = {
"input",
"Default",
"/etc/varnish/publications-begin.txt",
"/etc/varnish/include-clusters.vcl",
"/etc/varnish/include-clusters.vcl",
"/etc/varnish/include-clusters.vcl",
"/etc/varnish/include-clusters.vcl",
"/etc/varnish/include-clusters.vcl",
"/etc/varnish/include-clusters.vcl",
"/etc/varnish/publications-end.txt",
"/etc/varnish/include-clusters.vcl",
"/etc/varnish/include-clusters.vcl",
"/etc/varnish/include-clus
(gdb) bt
#0 0x0000003abde30155 in raise () from /lib64/libc.so.6
#1 0x0000003abde31bf0 in abort () from /lib64/libc.so.6
#2 0x000000000041af57 in pan_ic (func=<value optimized out>, file=<value
optimized out>, line=<value optimized out>, cond=<value optimized out>,
err=64, xxx=<value optimized out>) at cache_panic.c:317
#3 0x00000000004200c5 in VRT_r_req_backend_healthy (sp=0x2aabebe4f008) at
cache_vrt.c:663
#4 0x00002aabeb5071c8 in ?? ()
#5 0x000000005cc96bd0 in ?? ()
#6 0x00002aabebe4f008 in ?? ()
#7 0x00002aabebe4f008 in ?? ()
#8 0x000000000041f416 in VCL_deliver_method (sp=0x2aabebe4f008) at
../../include /vcl_returns.h:59
#9 0x000000000040f77a in cnt_deliver (sp=0x2aabebe4f008) at
cache_center.c:166
#10 0x00000000004104ba in CNT_Session (sp=0x5cc94b20) at steps.h:42
#11 0x000000000041c4f2 in wrk_do_cnt_sess (w=0x5cc96bd0, priv=<value
optimized out>) at cache_pool.c:398
#12 0x000000000041bbbf in wrk_thread (priv=0x2ad8203331f0) at
cache_pool.c:310
#13 0x0000003abe6062f7 in start_thread () from /lib64/libpthread.so.0
#14 0x0000003abded1b6d in clone () from /lib64/libc.so.6
}}}
--
Ticket URL: <http://varnish.projects.linpro.no/ticket/508#comment:2>
Varnish <http://varnish.projects.linpro.no/>
The Varnish HTTP Accelerator
More information about the varnish-bugs
mailing list