Varnish returning 503s for Googlebot requests (Bug #813?)

[Mattias Geniar]

Hi Ronan,

Not sure if you've managed to test this yet, but Google seem to run with
"Accept-Encoding: gzip". Perhaps there's a problem serving the
compressed version, whereas your manual wget's don't use this
accept-encoding?

Regards,
Mattias

-----Original Message-----
From: varnish-misc-bounces at varnish-cache.org
[mailto:varnish-misc-bounces at varnish-cache.org] On Behalf Of Ronan
Mullally
Sent: zaterdag 5 maart 2011 10:48
To: varnish-misc at varnish-cache.org
Subject: Varnish returning 503s for Googlebot requests (Bug #813?)

Hi,

I'm a varnish noob.  I've only just started rolling out a cache in front
of a VBulletin site running Apache that is currently using pound for
load
balancing.

I'm running 2.1.5 on a debian lenny box.  Testing is going well, apart
from one problem.  The site runs VBSEO to generate sitemap files.
Without excpetion, every time Googlebot tries to request these files
Varnish returns a 503:

 66.249.66.246 - - [05/Mar/2011:09:33:53 +0000] "GET
http://www.sitename.net/sitemap_151.xml.gz HTTP/1.1" 503 419 "-"
   "Mozilla/5.0 (compatible; Googlebot/2.1;
+http://www.google.com/bot.html)"

I can request these files via wget direct from the backend as well as
direct from varnish without a problem:

 --2011-03-05 09:23:39--  http://www.sitename.net/sitemap_362.xml.gz

 HTTP request sent, awaiting response...
   HTTP/1.1 200 OK
   Server: Apache
   Content-Type: application/x-gzip
   Content-Length: 130283
   Date: Sat, 05 Mar 2011 09:23:38 GMT
   X-Varnish: 1282440127
   Age: 0
   Via: 1.1 varnish
   Connection: keep-alive
 Length: 130283 (127K) [application/x-gzip]
 Saving to: `/dev/null'

 2011-03-05 09:23:39 (417 KB/s) - `/dev/null' saved [130283/130283]

I've reverted back to default.vcl, the only changes being to define my
own
backends.  Varnishlog output is below.  Having googled a bit the only
thing I've found is bug #813, but that was apparently fixed prior to
2.1.5.  Am I missing something obvious?


-Ronan


Varnishlog output

   18 ReqStart     c 66.249.66.246 63009 1282436348
   18 RxRequest    c GET
   18 RxURL        c /sitemap_362.xml.gz
   18 RxProtocol   c HTTP/1.1
   18 RxHeader     c Host: www.sitename.net
   18 RxHeader     c Connection: Keep-alive
   18 RxHeader     c Accept: */*
   18 RxHeader     c From: googlebot(at)googlebot.com
   18 RxHeader     c User-Agent: Mozilla/5.0 (compatible; Googlebot/2.1;
+http://www.google.com/bot.html)
   18 RxHeader     c Accept-Encoding: gzip,deflate
   18 RxHeader     c If-Modified-Since: Sat, 05 Mar 2011 08:40:46 GMT
   18 VCL_call     c recv
   18 VCL_return   c lookup
   18 VCL_call     c hash
   18 VCL_return   c hash
   18 VCL_call     c miss
   18 VCL_return   c fetch
   18 Backend      c 40 sitename sitename1
   40 TxRequest    b GET
   40 TxURL        b /sitemap_362.xml.gz
   40 TxProtocol   b HTTP/1.1
   40 TxHeader     b Host: www.sitename.net
   40 TxHeader     b Accept: */*
   40 TxHeader     b From: googlebot(at)googlebot.com
   40 TxHeader     b User-Agent: Mozilla/5.0 (compatible; Googlebot/2.1;
+http://www.google.com/bot.html)
   40 TxHeader     b Accept-Encoding: gzip,deflate
   40 TxHeader     b X-Forwarded-For: 66.249.66.246
   40 TxHeader     b X-Varnish: 1282436348
   40 RxProtocol   b HTTP/1.1
   40 RxStatus     b 200
   40 RxResponse   b OK
   40 RxHeader     b Date: Sat, 05 Mar 2011 09:17:37 GMT
   40 RxHeader     b Server: Apache
   40 RxHeader     b Content-Length: 130327
   40 RxHeader     b Content-Encoding: gzip
   40 RxHeader     b Vary: Accept-Encoding
   40 RxHeader     b Content-Type: application/x-gzip
   18 TTL          c 1282436348 RFC 10 1299316657 0 0 0 0
   18 VCL_call     c fetch
   18 VCL_return   c deliver
   18 ObjProtocol  c HTTP/1.1
   18 ObjStatus    c 200
   18 ObjResponse  c OK
   18 ObjHeader    c Date: Sat, 05 Mar 2011 09:17:37 GMT
   18 ObjHeader    c Server: Apache
   18 ObjHeader    c Content-Encoding: gzip
   18 ObjHeader    c Vary: Accept-Encoding
   18 ObjHeader    c Content-Type: application/x-gzip
   18 FetchError   c straight read_error: 0
   40 Fetch_Body   b 4 4294967295 1
   40 BackendClose b sitename1
   18 VCL_call     c error
   18 VCL_return   c deliver
   18 VCL_call     c deliver
   18 VCL_return   c deliver
   18 TxProtocol   c HTTP/1.1
   18 TxStatus     c 503
   18 TxResponse   c Service Unavailable
   18 TxHeader     c Server: Varnish
   18 TxHeader     c Retry-After: 0
   18 TxHeader     c Content-Type: text/html; charset=utf-8
   18 TxHeader     c Content-Length: 419
   18 TxHeader     c Date: Sat, 05 Mar 2011 09:17:38 GMT
   18 TxHeader     c X-Varnish: 1282436348
   18 TxHeader     c Age: 1
   18 TxHeader     c Via: 1.1 varnish
   18 TxHeader     c Connection: close
   18 Length       c 419
   18 ReqEnd       c 1282436348 1299316657.660784483
1299316658.684726000 0.478523970 1.023897409 0.000044107
   18 SessionClose c error
   18 StatSess     c 66.249.66.246 63009 6 1 5 0 0 4 2984 32012


_______________________________________________
varnish-misc mailing list
varnish-misc at varnish-cache.org
http://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc