Different cache policy for googlebot

Poul-Henning Kamp phk at phk.freebsd.dk
Thu Dec 2 11:33:12 CET 2010

In message <8A96548A-C3BE-43B9-9EA1-BE80B49D56D9 at collaborativebusiness.co.uk>, 
David Turner writes:

>We have slowed Googlebot down, and maybe that's the best solution
>at the end of the day. My worry with that is that the site is growing
>quickly so by slowing Google down, it has an older view of the whole
>site whereas my proposal is to let it keep coming but let it hold
>an older version of each page.

You only need to slow it on cache misses ?

Poul-Henning Kamp       | UNIX since Zilog Zeus 3.20
phk at FreeBSD.ORG         | TCP/IP since RFC 956
FreeBSD committer       | BSD since 4.3-tahoe    
Never attribute to malice what can adequately be explained by incompetence.

More information about the varnish-misc mailing list