Let GoogleBot Crawl full content, reverse DNS lookup
rlane at ahbelo.com
Mon Mar 7 15:58:08 CET 2011
I am looking into supporting Google’s “First Click Free for Web Search”. I need to allow the GoogleBots to index the full content of my sites but still maintain the Registration wall for everyone else. Google suggests that you detect there GoogleBots by reverse DNS lookup of the requesters IP.
Google Desc: http://www.google.com/support/webmasters/bin/answer.py?answer=80553
Has anyone done DNS lookups via VCL to verify access to content or to cache content?
-------------- next part --------------
An HTML attachment was scrubbed...
More information about the varnish-misc