Assert error in VRT_r_req_backend_healthy()

[Simon Lyall]

I have just upgraded a couple of my caches from version 2.1 to 3.0.3 ( 
using centos5 RPMs).

I notice that I'm getting a lot of crashes (about twice an hour on a 
lightly loaded machine) when previously I had none.

My crashes all look like:

Nov  7 20:21:13 harvest varnishd[3290]: Child (5139) died signal=6
Nov  7 20:21:13 harvest varnishd[3290]: Child (5139) Panic message: 
Assert error in VRT_r_req_backend_healthy(), cache_vrt_var.c line 539: 
Condition((sp->director) != NULL) not true. thread = (cache-worker) ident 
= Linux,2.6.18-308.16.1.el5,x86_64,-sfile,-smalloc,-hcritbit,epoll 
Backtrace:   0x42dc76: /usr/sbin/varnishd [0x42dc76]   0x438803: 
/usr/sbin/varnishd(VRT_r_req_backend_healthy+0xd3) [0x438803] 
0x2aaaba6037dc: ./vcl.pi3D54nW.so [0x2aaaba6037dc]   0x434626: 
/usr/sbin/varnishd(VCL_deliver_method+0x46) [0x434626]   0x4179b3: 
/usr/sbin/varnishd [0x4179b3]   0x418938: 
/usr/sbin/varnishd(CNT_Session+0x8a8) [0x418938]   0x42f706: 
/usr/sbin/varnishd [0x42f706]   0x3acc80677d: /lib64/libpthread.so.0 
[0x3acc80677d]   0x3acb8d3c1d: /lib64/libc.so.6(clone+0x6d) [0x3acb8d3c1d] 
sp = 0x2aaac7924008 {   fd = 167, id = 167, xid = 121392249,   client = 
178.128.200.204 64256,   step = STP_PREPRESP,   handling = deliver, 
restarts = 0, esi_level = 0   flags =    bodystatus = 3   ws = 
0x2aaac7924080 {      id = "sess
Nov  7 20:21:13 harvest varnishd[3290]: child (5444) Started
Nov  7 20:21:13 harvest varnishd[3290]: Child (5444) said Child starts
Nov  7 20:21:13 harvest varnishd[3290]: Child (5444) said SMF.s0 mmap'ed 
262144000 bytes of 262144000


Searching Around I came across this thread:

http://www.gossamer-threads.com/lists/varnish/bugs/25391

but couldn't find an actual bug that matched to it.


The servers are on the far side of the world from the origin so I would 
expect the odd heath failure, they are faily lightly loaded though ( only 
a couple of hundred hits/second max) My vcl_deliver looks like:

-------------------
# vcl_deliver Body
#

         if (resp.http.magicmarker) {

                 /* Remove the magic marker */
                 unset resp.http.magicmarker;

                 /* By definition we have a fresh object */
                 set resp.http.originalage = resp.http.Age;
                 set resp.http.Age = "0";
         }


# Create temp variable header
if (req.backend.healthy) {
         set resp.http.Backend-Healthy = "yes" ;
} else {
         set resp.http.Backend-Healthy = "no" ;
}

# Debugging
set resp.http.Cache-Info = "Object-Age=" + resp.http.originalage + " hits=" + obj.hits + " Cache-Host=" + server.hostname + " Backend-Host=" + req.backend  + "healthy=" + resp.http.Backend-Healthy ;

# Tidy some headers
unset resp.http.X-Varnish;
unset resp.http.Via;
unset resp.http.originalage;
unset resp.http.Age;
unset resp.http.Backend-Healthy;
-------------------

My backend probes mostly look like:

-----
backend xx_xxx121 {
       .host = "xx.xx.xx.xx";
       .host_header = "www.xx.xx";
       .port = "80";
       .probe = {
         .url = "/";
         .timeout = 5s;
         .interval = 10s;
         .window = 10;
         .threshold = 6;
         .initial = 5;
       }
}
----

-- 
Simon Lyall  |  Very Busy  |  Web: http://www.darkmere.gen.nz/
"To stay awake all night adds a day to your life" - Stilgar | eMT.