503 returning FetchError c http first read error: -1 0

[Adam McManus]

I could use some help troubleshooting why users sometimes receive a 503
error from Varnish. It does not happen every time, but it is fairly
reproducible on the below example (http://gowalla.com/users/keeg). The
backends are all configured as follows:

backend server1 {
  .host = "<internal_ip>";
  .port = "<internal_port>";
  .connect_timeout = 30s;
  .first_byte_timeout = 30s;
  .between_bytes_timeout = 30s;
}

The director configuration is:

director rr_director round-robin {
  { .backend = server1; }
  { .backend = server2; }
  { .backend = server3; }
  { .backend = server4; }
  { .backend = server5; }
  { .backend = server6; }
  { .backend = server7; }
  { .backend = server8; }
  { .backend = server9; }
}

Here is the varnishlog output of a 503 response. I've searched everywhere
for information on the FetchError provided, but I can't find anything that
seems to apply. Also, the request is not hitting any of the timeouts listed
in the backend configuration. The 503 response usually returns in less than
a second.

   27 SessionOpen  c 127.0.0.1 33235 :8080
   27 ReqStart     c 127.0.0.1 33235 1983217687
   27 RxRequest    c GET
   27 RxURL        c /users/keeg
   27 RxProtocol   c HTTP/1.0
   27 RxHeader     c X-Real-IP: 24.153.226.54
   27 RxHeader     c X-Forwarded-For: 24.153.226.54
   27 RxHeader     c Host: gowalla.com
   27 RxHeader     c Connection: close
   27 RxHeader     c Referer: http://gowalla.com/users/keeg
   27 RxHeader     c Cache-Control: max-age=0
   27 RxHeader     c Accept:
application/xml,application/xhtml+xml,text/html;q=0.9,text/plain;q=0.8,image/png,*/*;q=0.5
   27 RxHeader     c User-Agent: Mozilla/5.0 (Macintosh; U; Intel Mac OS X
10_6_6; en-US) AppleWebKit/534.13 (KHTML, like Gecko) Chrome/9.0.597.102
Safari/534.13
   27 RxHeader     c Accept-Encoding: gzip,deflate,sdch
   27 RxHeader     c Accept-Language: en-US,en;q=0.8
   27 RxHeader     c Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.3
   27 RxHeader     c If-None-Match: "bf9753b17d60d033e7654ea22abfcbd8"
   27 VCL_call     c recv pass
   27 VCL_call     c hash hash
   27 VCL_call     c pass pass
   27 Backend      c 29 rr_director server9
   27 FetchError   c http first read error: -1 0 (Success)
   27 VCL_call     c error deliver
   27 VCL_call     c deliver deliver
   27 TxProtocol   c HTTP/1.1
   27 TxStatus     c 503
   27 TxResponse   c Service Unavailable
   27 TxHeader     c Server: Varnish
   27 TxHeader     c Retry-After: 0
   27 TxHeader     c Content-Type: text/html; charset=utf-8
   27 TxHeader     c Content-Length: 419
   27 TxHeader     c Date: Thu, 17 Feb 2011 20:44:21 GMT
   27 TxHeader     c X-Varnish: 1983217687
   27 TxHeader     c Age: 1
   27 TxHeader     c Via: 1.1 varnish
   27 TxHeader     c Connection: close
   27 Length       c 419
   27 ReqEnd       c 1983217687 1297975461.143035889 1297975461.773808956
0.000031948 0.630686998 0.000086069

I would appreciate any insights anyone might have.

Thanks,

--
Adam McManus
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.varnish-cache.org/lists/pipermail/varnish-misc/attachments/20110217/5bb05602/attachment.html>