Intermittent 503 errors

[Sam Crawford]

Morning all,

Once every few days we get a report of a user with a nice big "503
Service Unavailable" on a webpage that refreshes itself every 30
seconds.

The webserver that serves the content appears to be up continuously,
although it could well be garbage collecting (it's GlassFish - I need
to enable GC logs to capture this activity). Also, our backend connect
timeout in our Varnish VCL is currently set to 1s, which could be
problematic.

In order to help confirm my suspicions would someone mind casting
their eyes over the log below? I'm interested to know what caused
Varnish to report the 503 - was it a plain connection refused? Did the
connection to the backend timeout? I can't tell myself from this log.

Thanks,

Sam


   14 SessionOpen  c 10.99.1.15 4679 10.98.13.28:7070
   14 ReqStart     c 10.99.1.15 4679 1053467939
   14 RxRequest    c GET
   14 RxURL        c /some/uri/goes/here
   14 RxProtocol   c HTTP/1.1
   14 RxHeader     c Accept: image/gif, image/x-xbitmap, image/jpeg,
image/pjpeg, application/vnd.ms-excel, application/vnd.ms-powerpoint,
application/msword,
 application/x-shockwave-flash, application/xaml+xml,
application/vnd.ms-xpsdocument, application/x-ms-xbap, applicati
   14 RxHeader     c Accept-Language: en-gb
   14 RxHeader     c UA-CPU: x86
   14 RxHeader     c Accept-Encoding: gzip, deflate
   14 RxHeader     c User-Agent: Mozilla/4.0 (compatible; MSIE 7.0;
Windows NT 5.1; .NET CLR 1.1.4322; .NET CLR 2.0.50727; .NET CLR
3.0.04506.30; .NET CLR 3.0
.04506.648; .NET CLR 3.5.21022; InfoPath.2; .NET CLR 3.0.4506.2152;
.NET CLR 3.5.30729)
   14 RxHeader     c Host: intranet.company.com
   14 RxHeader     c Connection: Keep-Alive
   14 RxHeader     c Cookie: amlbcookie=01
   14 VCL_call     c recv
   14 VCL_return   c pass
   14 VCL_call     c pass
   14 VCL_return   c pass
   14 Backend      c 27 localhost localhost
   27 TxRequest    b GET
   27 TxURL        b /some/uri/goes/here
   27 TxProtocol   b HTTP/1.1
   27 TxHeader     b Accept: image/gif, image/x-xbitmap, image/jpeg,
image/pjpeg, application/vnd.ms-excel, application/vnd.ms-powerpoint,
application/msword,
 application/x-shockwave-flash, application/xaml+xml,
application/vnd.ms-xpsdocument, application/x-ms-xbap, applicati
   27 TxHeader     b Accept-Language: en-gb
   27 TxHeader     b UA-CPU: x86
   27 TxHeader     b User-Agent: Mozilla/4.0 (compatible; MSIE 7.0;
Windows NT 5.1; .NET CLR 1.1.4322; .NET CLR 2.0.50727; .NET CLR
3.0.04506.30; .NET CLR 3.0
.04506.648; .NET CLR 3.5.21022; InfoPath.2; .NET CLR 3.0.4506.2152;
.NET CLR 3.5.30729)
   27 TxHeader     b Host: intranet.company.com
   27 TxHeader     b Cookie: amlbcookie=01
   27 TxHeader     b Accept-Encoding: gzip
   27 TxHeader     b X-Varnish: 1053467939
   27 TxHeader     b X-Forwarded-For: 10.99.1.15
   27 BackendClose b localhost
   14 VCL_call     c error
   14 VCL_return   c deliver
   14 Length       c 466
   14 VCL_call     c deliver
   14 VCL_return   c deliver
   14 TxProtocol   c HTTP/1.1
   14 TxStatus     c 503
   14 TxResponse   c Service Unavailable
   14 TxHeader     c Server: Varnish
   14 TxHeader     c Retry-After: 0
   14 TxHeader     c Content-Type: text/html; charset=utf-8
   14 TxHeader     c Content-Length: 466
   14 TxHeader     c Date: Thu, 30 Jul 2009 09:57:46 GMT
   14 TxHeader     c X-Varnish: 1053467939
   14 TxHeader     c Age: 0
   14 TxHeader     c Via: 1.1 varnish
   14 TxHeader     c Connection: close
   14 ReqEnd       c 1053467939 1248947866.962788105
1248947866.963104010 0.000128031 0.000283957 0.000031948
   14 SessionClose c error