Varnish and time out on backend (first_byte_timeout).

Mickaël GERVAIS mgervais at agaetis.fr
Fri Jan 14 10:29:54 CET 2011


Hi,

It's me again, apparently, my configuration doesn't work. When the backend
is down, vcl_error is called, but it's not due to a timeout, so I've a 503
error then I call restart. But even if I've specified a grace mode to 3h in
fetch my object is available for at least 2 min... Does this trick works
(in order to handle timeout) with grace mode when backend is down? My
config is attached. The request is restared 4 times (see config) but objet
is not retrieve from cache...

Thanks.

Mickael

P.S: Sorry for my english...






On Wed, 12 Jan 2011 16:08:13 +0100, Mickaël GERVAIS <mgervais at agaetis.fr>
wrote:
> Thanks a lot!! Apparently it works... (I've taken my hearplugs...)
> Here is my config is somebody needs it:
> 
> backend fake {
>      .host = "xxxxxxxxx";
>      .port = "80";
>      .probe = {
>         .url = "/fake.html";
>         .interval = 60s;
>         .timeout = 0.1s;
>         .window = 1;
>         .threshold = 1;
>         .initial = 1;
>      }
> }
> 
> sub vcl_recv {
> [...]
>    if ( req.http.magicmarker && req.http.magicmarker == "fake" ) {
>        unset req.http.magicmarker;
>        set req.backend = fake;
>    } else {
>        set req.backend = yyyy;
>    }
> [...]
> }
> 
> sub vcl_error {
>      log "[Error ]  ( ) " req.url "(Status: " obj.status ", Restarts: "
> req.restarts ")";
>      if (obj.status == 503 && req.restarts < 5) {
>         log "--- Restart url: " req.url "(Status: " obj.status ",
Restarts:
> " req.restarts ")";
>         set obj.http.X-Restarts = req.restarts;
>         if ( req.restarts == 0 ){
>                 log "--- First restart add fake.";
>                 set req.http.magicmarker = "fake";
>         }
>         restart;
>     }
> }
> 
> On Wed, 12 Jan 2011 14:31:58 +0100, Kristian Lyngstol
> <kristian at varnish-software.com> wrote:
>> On Wed, Jan 12, 2011 at 11:34:37AM +0100, Mickaël GERVAIS wrote:
>>> If a timeout occurs (first_byte_timeout reached) the function vcl_error
>>> is
>>> called, I'd like to use the saint mode to retreive the response from
the
>>> cache, but saint mode is only avaliable on beresp. 
>>> 
>>> Is there a way to tell varnish use a dirty object from the cache? Maybe
>>> is
>>> not the correct way to handle this kind of error. 
>> 
>> You are correct - that is a weakness. I have a nasty hack, though.
>> 
>> 1. Declare a second, bogus backend which will always be sick.
>> 2. In vcl_error if restarts is 0, set a magic marker and restart.
>> 3. Look for the magic marker in vcl_recv - if it's present, tell Varnish
> to
>>    use the bogus backend. Grace will then kick in because that backend
is
>>    marked as sick.
>> 4. If the object exists in cache (graced) - it will be used. Otherwise,
> you
>>    will hit vcl_error again. (Thus the check of req.restarts in step 2).
>> 
>> It's a nasty, yet brilliant hack, if I might say so myself ;)
>> 
>> It adds latency and doesn't utilize saintmode, but it gets the job done
> in
>> a way that will also make little children cry.
>> 
>> - Kristian

-- 
:::::::::::::::::::::::::::::::::::::::::::::::
MICKAL GERVAIS

Agaetis
10 allée Evariste Galois
63 000 Clermont-Ferrand

Courriel : mgervais at agaetis.fr
Téléphone : 04 73 44 56 51
Portable : 06 82 35 52 82
Site : http://www.agaetis.fr
:::::::::::::::::::::::::::::::::::::::::::::::
-------------- next part --------------
#################
# Back-end. #
#################
backend www {
    .host = "xxxxxxxxx";
    .port = "8081";
    .connect_timeout = 10s;
    .first_byte_timeout = 30s; 
    .between_bytes_timeout = 1s;
    .probe = {
        .request =
            "GET /loader.gif HTTP/1.1"
            "Host: xxxxxxxxx:8081"
            "Connection: close";
        .interval = 2s;
        .timeout = 1s;
        .window = 20;
        .threshold = 19;
        .initial = 19;
    }
}

#######################################
# Fake back-end which is always sick. #
#######################################
backend fake {
    .host = "xxxxxxxxx";
    .port = "8081";
    .probe = {
        .url = "/fake.html";
        .interval = 60s;
        .timeout = 0.1s;
        .window = 2;
        .threshold = 2;
        .initial = 0;
    }
}

#########################################################################################################################
# Called at the beginning of a request, after the complete request has been received and parsed.                        #
# Its purpose is to decide whether or not to serve the request, how to do it, and, if applicable, which backend to use. #
#########################################################################################################################
sub vcl_recv {
    # Section to purge an URL.
    if ( req.request == "PURGE" ) {
        purge("req.url ~ " req.url);
        error 200 "Purged";
    }

    # Add an unique header containing the client address
    unset req.http.X-Forwarded-For;
    set req.http.X-Forwarded-For = client.ip;

    if (req.http.host ~ "beta.(xxxxxxxxxx).fr$") {
        # Redirection for mobile site.
        if ( req.http.user-agent ~ "^((Fly|FLY|HTC|LG|MAUI|MOT|SEC|SIE)|(.*(ACS-NF|Android|Alcatel|Amoi|BENQ|BenQ|BlackBerry|Cellphone|DoCoMo|Ericsson|Hutchison|iPAQ|iPhone|MIDP|Mitsu|mobile'|Mobile'|Motorola|Nokia|Palm|Panasonic|PHILIPS|portalmmm|SAGEM|Samsung|SAMSUNG|Sanyo|SANYO|SCH\-|Sendo|SHARP|SmartPhone|Smartphone|Symbian\ OS|SymbianOS|Toshiba|UP\.Browser|Vodafone|Windows\ CE)))" && 
             req.url == "/" ) {
            set req.http.mobilehost = regsub(req.http.host, "^beta\.(.+)$", "http://m.\1");
            log "[Receive] " req.url " Redirection to mobile URL. (" req.http.mobilehost ")";
            error 750 req.http.mobilehost;
        }
        
        if ( req.http.magicmarker && req.http.magicmarker == "fake" ) {
            unset req.http.magicmarker;
            set req.backend = fake;
        } else {
           set req.backend = www;
        }
    }
    
    # Force cache for static files even if a cookie exists.
    if ( req.url ~ "\.(js|css|jpg|JPG|jpeg|JPEG|png|gif|swf|ico)(\?|$)" ) {
        remove req.http.cookie;
    }

    if ( req.request != "GET" && 
         req.request != "HEAD" ) {
        log "[Receive] " req.url " not cached.";
        return (pass);
    }
    
    if ( req.url ~ ".*/(pdf|acces|ajax|.*\.shtml|json-rpc|captcha\.jpg|balancer-manager).*" ) {
        log "[Receive] " req.url " not cached.";
        return (pass);
    }

    if ( req.backend.healthy ) {
        set req.grace = 15s;
        log "[Receive] " req.url "(Back-end " req.backend " healthy, Grace: " req.grace ")";    
    } else {
        set req.grace = 1m; 
        log "[Receive] " req.url "(Back-end " req.backend " not healthy, Grace: " req.grace ")"; 
    }

    return (lookup);
}

#############################################################################################################
# Called upon entering pipe mode. In this mode, the request is passed on to the backend, and any            #
# further data from either client or backend is passed on unaltered until either end closes the connection. #
#############################################################################################################
sub vcl_pipe {
    log "[Pipe   ] " req.url;
}

#############################################################################################
# Called upon entering pass mode. In this mode, the request is passed on to the backend,    #
# and the backend’s response is passed on to the client, but is not entered into the cache. #
# Subsequent requests sub-mitted over the same client connection are handled normally.      #
#############################################################################################
sub vcl_pass {
    log "[Pass   ] " req.url;
}

####################################################################################################
# Use req.hash += req.http.Cookie or similar to include the Cookie HTTP header in the hash string. #
####################################################################################################
sub vcl_hash {
#    log "[Hash   ] " req.url;
}
#################################################################################
# Called after a cache lookup if the requested document was found in the cache. #
#################################################################################
sub vcl_hit {
    if ( obj.cacheable ) {
        log "[Hit    ] " req.url " (Cacheable: YES)";                
    } else {
        log "[Hit    ] " req.url " (Cacheable: NO)";
    }
}

###########################################################################################################################
# Called after a cache lookup if the requested document was not found in the cache.                                       #
# Its purpose is to decide whether or not to attempt to retrieve the document from the backend, and which backend to use. #
###########################################################################################################################
sub vcl_miss {
    log "[Miss   ] " req.url;

    if ( req.request == "PURGE" ) {
        error 200 "Not in cache";
    }
}

#############################################################################
# Called after a document has been successfully retrieved from the backend. #
#############################################################################
sub vcl_fetch {
    if ( beresp.status == 500 ) {
        set beresp.saintmode = 20s;
        log "[Fetch  ] " bereq.url " (Saint: 20s)";
        restart;
    }

    set beresp.grace = 3h;
    
    # These status code 404 should always pass through and never cache.
    if ( beresp.status == 404 ) {
        log "[Fetch  ] " bereq.url " (Status:" beresp.status " not cached -> Pass.)";     
        set beresp.http.X-Cacheable = "NO: beresp.status";
        set beresp.http.X-Cacheable-status = beresp.status;
        return (pass);
    }
    
    if( beresp.cacheable ) {
        log "[Fetch  ] " bereq.url " (Grace:" beresp.grace ", TTL:" beresp.ttl ", Status:" beresp.status ", Cacheable: YES)";     
    } else {
        log "[Fetch  ] " bereq.url " (Grace:" beresp.grace ", TTL:" beresp.ttl ", Status:" beresp.status ", Cacheable: NO)";
    }
}

#############################################################
# Called before a cached object is delivered to the client. #
#############################################################
sub vcl_deliver {
    # Add cache hit data
    if ( obj.hits > 0 ) {
        set resp.http.X-Cache = "HIT";
        set resp.http.X-Cache-Hits = obj.hits;
    } else {
        set resp.http.X-Cache = "MISS";
    }

    set resp.http.X-BackEnd = req.backend;
    set resp.http.X-Restarts = req.restarts;
}

sub vcl_error {
    # Redirect, error sent from vcl_receive.
    if ( obj.status == 750 ) {
        set obj.http.Location = obj.response;
        set obj.status = 302;
        return (deliver);
    }

    log "[Error  ] " req.url " (Status: " obj.status ", Restarts: " req.restarts ")";
     
    if ( obj.status == 503 && req.restarts < 5 ) {
        set obj.http.X-Restarts = req.restarts;
        if ( req.restarts == 0 ){
            set req.http.magicmarker = "fake";
        }
        restart;
    }

    set obj.http.Content-Type = "text/html; charset=utf-8";

    if (req.http.host ~ "beta.(xxxxxxxxxxxxxx).fr$") {
        synthetic {"<?xml version="1.0" encoding="utf-8"?>
            <!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN"
            "http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">
            <html>
                <head>
                    <title>"} obj.status " " obj.response {"</title>
                </head>
                <body>
                    <h1>Error "} obj.status " " obj.response {"</h1>
                    <p>"} obj.response {"</p>
                </body>
            </html>"};
    }
    return (deliver);
}


More information about the varnish-misc mailing list