RFC: new vcl_lookup{} proposal

[Martin Blix Grydeland]

As we are moving more logic from varnishd into VCL, I believe some
rethinking with how we deal with the default_vcl logic might be in order.
This to make it easier to make the easy VCL changes and keep the default
logic around still.

A common vcl error I have observed is where you want to match on something
to perform an action, and then stop further processing on this request so
further rules won't match on the url. E.g.:

sub vcl_fetch {
    if (req.url ~ "\.(gif|jpg|css") {
        # Cache our static resources "forever"
        set beresp.ttl = 180d;
        return (deliver);
    }


}

This will ofc bypass any set-cookie checks performed by the default
vcl_fetch logic. Also the explicit return(deliver) prevents any further
application level rules that may be applied later, which is convenient. But
the problem comes if for some reason one of these static resources should
actually return a set-cookie header.

So the correct thing to do in this case

If we changed how the default vcl is structured into having all of the
logic in separate subroutines, we would in cases li

On Mon, Sep 17, 2012 at 9:33 AM, Poul-Henning Kamp <phk at phk.freebsd.dk>wrote:

>
> Right now we go directly from vcl_hash{} to either vcl_hit{}, vcl_miss{}
> or vcl_pass{} depending on the outcome, and that means, amongst other
> things, that a VCL implementation of "PURGE" as to add code to all those.
>
> Furthermore we have some semi-magic things going on behind the scenes,
> such as grace and saint modes.
>
> Assume:
>
> * Backend fetches will be done by a different thread than the one
>   processing the clients request.
>
> * vcl_hash{} always is followed by vcl_lookup{} and that the above
> mentioned
>   logic gets moved to VCL, rather than C code.
>
> * vcl_hit{} goes away.
>
> Then a plausible default vcl_lookup{} could then look like:
>
>         sub vcl_lookup {
>
>                 if (!obj) {
>                         // Miss:  Start a background fetch in another
> thread
>                         // We come back here, when the status of that fetch
>                         // is available. (ie: we go on the waiting list)
>                         return (miss);
>                 }
>
>                 if (obj.ttl <= 0s &&
>                     obj.ttl > -req.grace && obj.ttl > -obj.grace) {
>                         // We have a grace situation
>                         if (!obj.busy) {
>                                 // Nobody is fetching yet, so start a
>                                 // background fetch
>                                 background_fetch();
>                         }
>                         // NB: First client does not hang on the fetch
>                         return (deliver);
>                 }
>
>                 if (obj.ttl <= 0s) {
>                         // object is too old, fetch new one (see miss)
>                         return (miss);
>                 }
>
>                 return (deliver);
>         }
>
> Possible customizations:
>
> Purging:
>
>         if (req.request == "PURGE") {
>                 // We pressume access control was in vcl_recv{}
>                 return (purge);
>                 return (purge_all);
>         }
>
> Prefetching:
>
>         if (obj.ttl < 2s && !obj.busy) {
>                 // Close to deadline, pre-fetch
>                 background_fetch();
>         }
>
> Don't stream, only deliver complete objects:
>
>         if (!obj.complete) {
>                 wait_complete(obj);
>         }
>
> I'm seriously pondering if saintmode should become a vmod at the same
> time, in which case it would probably look something like:
>
>         sub vcl_lookup {
>                 if (obj.ttl <= 0s && !saint.ok(req.backend, req.url)) {
>                         return (deliviver)
>                 }
>         }
>
>         sub vcl_fetch {
>                 if (beresp.status >= 400) {
>                         saint.bad(bereq.backend, bereq.url);
>                 }
>         }
>
> --
> Poul-Henning Kamp       | UNIX since Zilog Zeus 3.20
> phk at FreeBSD.ORG         | TCP/IP since RFC 956
> FreeBSD committer       | BSD since 4.3-tahoe
> Never attribute to malice what can adequately be explained by incompetence.
>
> _______________________________________________
> varnish-dev mailing list
> varnish-dev at varnish-cache.org
> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-dev
>



-- 
Martin Blix Grydeland
Varnish Software AS
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/varnish-dev/attachments/20121001/1b03e79a/attachment.html>