Best practice for not caching content requested by crawlers

[Lasse Karstensen]

Lasse Karstensen:
[..]
> sub vcl_miss {
>     if (req.http.user-agent ~ "(?i)yandex|msnbot") {
>         return(pass);
>     }
> }
> You can probably use openddr/deviceatlas/$favorite_detectionengine to get
> better accuracy than this regex.

I took at look at some access logs and updated devicedetect.vcl a bit so
it has rudimentary bot detection:

    https://github.com/varnish/varnish-devicedetect/blob/master/devicedetect.vcl


-- 
Lasse Karstensen
Varnish Software AS
http://www.varnish-software.com/