backend probing vs grace on healthy settings

[ingard mevåg]

Hi

We recently experienced a problem with one of our websites running 
varnish 2.0.4 with grace enabled.

relevant config:

backend default {
           .host = "x.x.x.x";
           .port = "80";
           .probe = {
                   .timeout = 500 ms;
                   .interval = 5s;
                   .window = 10;
                   .threshold = 1;
                   .request =
                       "GET /index.php HTTP/1.1"
                       "Host: bladiblahdiblah.no"
                       "Connection: close"
                       "Accept-Encoding: text/html" ;
           }
     }
     sub vcl_recv {
             if (req.backend.healthy) {
                   set req.grace = 30s;
           } else {
                   set req.grace = 30m;
           }
   }
  The question is: When will the backend be marked sick? Is this after 
.interval X .window (50 secs) and in that case, what happens after the 
backend goes down and grace objects are served for 30 secs (as the 
backend is still marked as healthy). There seems to be 20 secs of 
"nothing" before the backend is decalred sick.

Did I enterpret this correctly? I was told in #varnish that the backend 
will be marked sick after .threshold of .window probes. This does not 
seem to be working for me atleast. As excerpt below shows, the backend 
isnt marked sick until all .window probes has been executed. However, it 
is marked as healthy after .threshold of .window probes after i start 
the backend server again.

Example:
--------------------------------------------------

debug.health
200 624    Backend default is Healthy
Current states  good: 10 threshold:  1 window: 10
Average responsetime of good probes: 0.001146
Oldest                                                    Newest
================================================================
4444444444444444444444444444444444444444444444444444444444444444 Good IPv4
XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX Good Xmit
SSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSS Good Shut
RRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRR Good Recv
HHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHH Happy

debug.health
200 624    Backend default is Healthy
Current states  good:  9 threshold:  1 window: 10
Average responsetime of good probes: 0.001146
Oldest                                                    Newest
================================================================
444444444444444444444444444444444444444444444444444444444444444- Good IPv4
XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX- Good Xmit
SSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSS- Good Shut
RRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRR- Good Recv
HHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHH- Happy

debug.health
200 624    Backend default is Healthy
Current states  good:  8 threshold:  1 window: 10
Average responsetime of good probes: 0.001146
Oldest                                                    Newest
================================================================
44444444444444444444444444444444444444444444444444444444444444-- Good IPv4
XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX-- Good Xmit
SSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSS-- Good Shut
RRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRR-- Good Recv
HHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHH-- Happy

debug.health
200 624    Backend default is Healthy
Current states  good:  7 threshold:  1 window: 10
Average responsetime of good probes: 0.001146
Oldest                                                    Newest
================================================================
4444444444444444444444444444444444444444444444444444444444444--- Good IPv4
XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX--- Good Xmit
SSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSS--- Good Shut
RRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRR--- Good Recv
HHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHH--- Happy

debug.health
200 624    Backend default is Healthy
Current states  good:  6 threshold:  1 window: 10
Average responsetime of good probes: 0.001146
Oldest                                                    Newest
================================================================
444444444444444444444444444444444444444444444444444444444444---- Good IPv4
XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX---- Good Xmit
SSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSS---- Good Shut
RRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRR---- Good Recv
HHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHH---- Happy

debug.health
200 624    Backend default is Healthy
Current states  good:  5 threshold:  1 window: 10
Average responsetime of good probes: 0.001146
Oldest                                                    Newest
================================================================
44444444444444444444444444444444444444444444444444444444444----- Good IPv4
XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX----- Good Xmit
SSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSS----- Good Shut
RRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRR----- Good Recv
HHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHH----- Happy

debug.health
200 624    Backend default is Healthy
Current states  good:  4 threshold:  1 window: 10
Average responsetime of good probes: 0.001146
Oldest                                                    Newest
================================================================
4444444444444444444444444444444444444444444444444444444444------ Good IPv4
XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX------ Good Xmit
SSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSS------ Good Shut
RRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRR------ Good Recv
HHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHH------ Happy

debug.health
200 624    Backend default is Healthy
Current states  good:  3 threshold:  1 window: 10
Average responsetime of good probes: 0.001146
Oldest                                                    Newest
================================================================
444444444444444444444444444444444444444444444444444444444------- Good IPv4
XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX------- Good Xmit
SSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSS------- Good Shut
RRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRR------- Good Recv
HHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHH------- Happy

debug.health
200 624    Backend default is Healthy
Current states  good:  2 threshold:  1 window: 10
Average responsetime of good probes: 0.001146
Oldest                                                    Newest
================================================================
44444444444444444444444444444444444444444444444444444444-------- Good IPv4
XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX-------- Good Xmit
SSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSS-------- Good Shut
RRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRR-------- Good Recv
HHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHH-------- Happy

debug.health
200 624    Backend default is Healthy
Current states  good:  1 threshold:  1 window: 10
Average responsetime of good probes: 0.001146
Oldest                                                    Newest
================================================================
4444444444444444444444444444444444444444444444444444444--------- Good IPv4
XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX--------- Good Xmit
SSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSS--------- Good Shut
RRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRR--------- Good Recv
HHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHH--------- Happy

debug.health
200 624    Backend default is Healthy
Current states  good:  1 threshold:  1 window: 10
Average responsetime of good probes: 0.001146
Oldest                                                    Newest
================================================================
4444444444444444444444444444444444444444444444444444444--------- Good IPv4
XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX--------- Good Xmit
SSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSS--------- Good Shut
RRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRR--------- Good Recv
HHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHH--------- Happy

debug.health
200 621    Backend default is Sick
Current states  good:  0 threshold:  1 window: 10
Average responsetime of good probes: 0.001146
Oldest                                                    Newest
================================================================
444444444444444444444444444444444444444444444444444444---------- Good IPv4
XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX---------- Good Xmit
SSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSS---------- Good Shut
RRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRR---------- Good Recv
HHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHH---------- Happy

debug.health
200 624    Backend default is Healthy
Current states  good:  1 threshold:  1 window: 10
Average responsetime of good probes: 0.001386
Oldest                                                    Newest
================================================================
444444444444444444444444444444444444444444444444444------------4 Good IPv4
XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX------------X Good Xmit
SSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSS------------S Good Shut
RRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRR------------R Good Recv
HHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHH------------H Happy