Let GoogleBot Crawl full content, reverse DNS lookup

[Mattias Geniar]

Hi,

I would look at the user agent to verify if it's a GoogleBot or not, as
that's more easily checked via VCL. All GoogleBots also adhere to the
correct User-Agent.
There really aren't that many users that spoof their User-Agent to gain
extra access.

Also keep in mind that serving GoogleBot different content than actual
users will get you penalties in SEO, eventually dropping your Google
ranking. Just, FYI.

Regards,
Mattias

From: varnish-misc-bounces at varnish-cache.org
[mailto:varnish-misc-bounces at varnish-cache.org] On Behalf Of Lane,
Richard
Sent: maandag 7 maart 2011 15:58
To: varnish-misc at varnish-cache.org
Subject: Let GoogleBot Crawl full content, reverse DNS lookup


I am looking into supporting Google's "First Click Free for Web Search".
I need to allow the GoogleBots to index the full content of my sites but
still maintain the Registration wall for everyone else. Google suggests
that you detect there GoogleBots by reverse DNS lookup of the requesters
IP. 

Google Desc:
http://www.google.com/support/webmasters/bin/answer.py?answer=80553

Has anyone done DNS lookups via VCL to verify access to content or to
cache content?

System Desc:
Varnish 2.1.4
RHEL 5-4
Apache 2.2x

- Richard