backend probing vs grace on healthy settings

[Ingard Mevåg]

Bump

On 12. mai. 2009, at 12.07, ingard mevåg <ingard at startsiden.no> wrote:

> Hi
>
> We recently experienced a problem with one of our websites running  
> varnish 2.0.4 with grace enabled.
>
> relevant config:
>
> backend default {
>           .host = "x.x.x.x";
>           .port = "80";
>           .probe = {
>                   .timeout = 500 ms;
>                   .interval = 5s;
>                   .window = 10;
>                   .threshold = 1;
>                   .request =
>                       "GET /index.php HTTP/1.1"
>                       "Host: bladiblahdiblah.no"
>                       "Connection: close"
>                       "Accept-Encoding: text/html" ;
>           }
>     }
>     sub vcl_recv {
>             if (req.backend.healthy) {
>                   set req.grace = 30s;
>           } else {
>                   set req.grace = 30m;
>           }
>   }
>  The question is: When will the backend be marked sick? Is this  
> after .interval X .window (50 secs) and in that case, what happens  
> after the backend goes down and grace objects are served for 30 secs  
> (as the backend is still marked as healthy). There seems to be 20  
> secs of "nothing" before the backend is decalred sick.
>
> Did I enterpret this correctly? I was told in #varnish that the  
> backend will be marked sick after .threshold of .window probes. This  
> does not seem to be working for me atleast. As excerpt below shows,  
> the backend isnt marked sick until all .window probes has been  
> executed. However, it is marked as healthy after .threshold  
> of .window probes after i start the backend server again.
>
> Example:
> --------------------------------------------------
>
> debug.health
> 200 624    Backend default is Healthy
> Current states  good: 10 threshold:  1 window: 10
> Average responsetime of good probes: 0.001146
> Oldest                                                    Newest
> ================================================================
> 4444444444444444444444444444444444444444444444444444444444444444  
> Good IPv4
> XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX  
> Good Xmit
> SSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSS  
> Good Shut
> RRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRR  
> Good Recv
> HHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHH Happy
>
> debug.health
> 200 624    Backend default is Healthy
> Current states  good:  9 threshold:  1 window: 10
> Average responsetime of good probes: 0.001146
> Oldest                                                    Newest
> ================================================================
> 444444444444444444444444444444444444444444444444444444444444444-  
> Good IPv4
> XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX-  
> Good Xmit
> SSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSS-  
> Good Shut
> RRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRR-  
> Good Recv
> HHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHH- Happy
>
> debug.health
> 200 624    Backend default is Healthy
> Current states  good:  8 threshold:  1 window: 10
> Average responsetime of good probes: 0.001146
> Oldest                                                    Newest
> ================================================================
> 44444444444444444444444444444444444444444444444444444444444444--  
> Good IPv4
> XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX--  
> Good Xmit
> SSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSS--  
> Good Shut
> RRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRR--  
> Good Recv
> HHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHH-- Happy
>
> debug.health
> 200 624    Backend default is Healthy
> Current states  good:  7 threshold:  1 window: 10
> Average responsetime of good probes: 0.001146
> Oldest                                                    Newest
> ================================================================
> 4444444444444444444444444444444444444444444444444444444444444---  
> Good IPv4
> XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX---  
> Good Xmit
> SSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSS---  
> Good Shut
> RRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRR---  
> Good Recv
> HHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHH--- Happy
>
> debug.health
> 200 624    Backend default is Healthy
> Current states  good:  6 threshold:  1 window: 10
> Average responsetime of good probes: 0.001146
> Oldest                                                    Newest
> ================================================================
> 444444444444444444444444444444444444444444444444444444444444----  
> Good IPv4
> XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX----  
> Good Xmit
> SSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSS----  
> Good Shut
> RRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRR----  
> Good Recv
> HHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHH---- Happy
>
> debug.health
> 200 624    Backend default is Healthy
> Current states  good:  5 threshold:  1 window: 10
> Average responsetime of good probes: 0.001146
> Oldest                                                    Newest
> ================================================================
> 44444444444444444444444444444444444444444444444444444444444-----  
> Good IPv4
> XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX-----  
> Good Xmit
> SSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSS-----  
> Good Shut
> RRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRR-----  
> Good Recv
> HHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHH----- Happy
>
> debug.health
> 200 624    Backend default is Healthy
> Current states  good:  4 threshold:  1 window: 10
> Average responsetime of good probes: 0.001146
> Oldest                                                    Newest
> ================================================================
> 4444444444444444444444444444444444444444444444444444444444------  
> Good IPv4
> XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX------  
> Good Xmit
> SSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSS------  
> Good Shut
> RRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRR------  
> Good Recv
> HHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHH------ Happy
>
> debug.health
> 200 624    Backend default is Healthy
> Current states  good:  3 threshold:  1 window: 10
> Average responsetime of good probes: 0.001146
> Oldest                                                    Newest
> ================================================================
> 444444444444444444444444444444444444444444444444444444444-------  
> Good IPv4
> XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX-------  
> Good Xmit
> SSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSS-------  
> Good Shut
> RRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRR-------  
> Good Recv
> HHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHH------- Happy
>
> debug.health
> 200 624    Backend default is Healthy
> Current states  good:  2 threshold:  1 window: 10
> Average responsetime of good probes: 0.001146
> Oldest                                                    Newest
> ================================================================
> 44444444444444444444444444444444444444444444444444444444--------  
> Good IPv4
> XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX--------  
> Good Xmit
> SSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSS--------  
> Good Shut
> RRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRR--------  
> Good Recv
> HHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHH-------- Happy
>
> debug.health
> 200 624    Backend default is Healthy
> Current states  good:  1 threshold:  1 window: 10
> Average responsetime of good probes: 0.001146
> Oldest                                                    Newest
> ================================================================
> 4444444444444444444444444444444444444444444444444444444---------  
> Good IPv4
> XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX---------  
> Good Xmit
> SSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSS---------  
> Good Shut
> RRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRR---------  
> Good Recv
> HHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHH--------- Happy
>
> debug.health
> 200 624    Backend default is Healthy
> Current states  good:  1 threshold:  1 window: 10
> Average responsetime of good probes: 0.001146
> Oldest                                                    Newest
> ================================================================
> 4444444444444444444444444444444444444444444444444444444---------  
> Good IPv4
> XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX---------  
> Good Xmit
> SSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSS---------  
> Good Shut
> RRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRR---------  
> Good Recv
> HHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHH--------- Happy
>
> debug.health
> 200 621    Backend default is Sick
> Current states  good:  0 threshold:  1 window: 10
> Average responsetime of good probes: 0.001146
> Oldest                                                    Newest
> ================================================================
> 444444444444444444444444444444444444444444444444444444----------  
> Good IPv4
> XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX----------  
> Good Xmit
> SSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSS----------  
> Good Shut
> RRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRR----------  
> Good Recv
> HHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHH---------- Happy
>
> debug.health
> 200 624    Backend default is Healthy
> Current states  good:  1 threshold:  1 window: 10
> Average responsetime of good probes: 0.001386
> Oldest                                                    Newest
> ================================================================
> 444444444444444444444444444444444444444444444444444------------4  
> Good IPv4
> XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX------------X  
> Good Xmit
> SSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSS------------S  
> Good Shut
> RRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRR------------R  
> Good Recv
> HHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHH------------H Happy
>