Let GoogleBot Crawl full content, reverse DNS lookup

Mattias Geniar mattias at nucleus.be
Mon Mar 7 16:05:08 CET 2011


I would look at the user agent to verify if it's a GoogleBot or not, as
that's more easily checked via VCL. All GoogleBots also adhere to the
correct User-Agent.
There really aren't that many users that spoof their User-Agent to gain
extra access.

Also keep in mind that serving GoogleBot different content than actual
users will get you penalties in SEO, eventually dropping your Google
ranking. Just, FYI.


From: varnish-misc-bounces at varnish-cache.org
[mailto:varnish-misc-bounces at varnish-cache.org] On Behalf Of Lane,
Sent: maandag 7 maart 2011 15:58
To: varnish-misc at varnish-cache.org
Subject: Let GoogleBot Crawl full content, reverse DNS lookup

I am looking into supporting Google's "First Click Free for Web Search".
I need to allow the GoogleBots to index the full content of my sites but
still maintain the Registration wall for everyone else. Google suggests
that you detect there GoogleBots by reverse DNS lookup of the requesters

Google Desc:

Has anyone done DNS lookups via VCL to verify access to content or to
cache content?

System Desc:
Varnish 2.1.4
RHEL 5-4
Apache 2.2x

- Richard 

More information about the varnish-misc mailing list