Intermittent 503 errors

[Sam Crawford]

I made a mistake in my original mail - there's a 5 second connect
timeout, not 1 second.

Also, I wonder if there may be some pooling/keep-alive mismatch
between Varnish and the webserver, whereby Varnish holds a stale
connections and tries to use it.

Suggestions welcome!

Thanks,

Sam


2009/7/30 Sam Crawford <samcrawford at gmail.com>:
> Morning all,
>
> Once every few days we get a report of a user with a nice big "503
> Service Unavailable" on a webpage that refreshes itself every 30
> seconds.
>
> The webserver that serves the content appears to be up continuously,
> although it could well be garbage collecting (it's GlassFish - I need
> to enable GC logs to capture this activity). Also, our backend connect
> timeout in our Varnish VCL is currently set to 1s, which could be
> problematic.
>
> In order to help confirm my suspicions would someone mind casting
> their eyes over the log below? I'm interested to know what caused
> Varnish to report the 503 - was it a plain connection refused? Did the
> connection to the backend timeout? I can't tell myself from this log.
>
> Thanks,
>
> Sam
>
>
>   14 SessionOpen  c 10.99.1.15 4679 10.98.13.28:7070
>   14 ReqStart     c 10.99.1.15 4679 1053467939
>   14 RxRequest    c GET
>   14 RxURL        c /some/uri/goes/here
>   14 RxProtocol   c HTTP/1.1
>   14 RxHeader     c Accept: image/gif, image/x-xbitmap, image/jpeg,
> image/pjpeg, application/vnd.ms-excel, application/vnd.ms-powerpoint,
> application/msword,
>  application/x-shockwave-flash, application/xaml+xml,
> application/vnd.ms-xpsdocument, application/x-ms-xbap, applicati
>   14 RxHeader     c Accept-Language: en-gb
>   14 RxHeader     c UA-CPU: x86
>   14 RxHeader     c Accept-Encoding: gzip, deflate
>   14 RxHeader     c User-Agent: Mozilla/4.0 (compatible; MSIE 7.0;
> Windows NT 5.1; .NET CLR 1.1.4322; .NET CLR 2.0.50727; .NET CLR
> 3.0.04506.30; .NET CLR 3.0
> .04506.648; .NET CLR 3.5.21022; InfoPath.2; .NET CLR 3.0.4506.2152;
> .NET CLR 3.5.30729)
>   14 RxHeader     c Host: intranet.company.com
>   14 RxHeader     c Connection: Keep-Alive
>   14 RxHeader     c Cookie: amlbcookie=01
>   14 VCL_call     c recv
>   14 VCL_return   c pass
>   14 VCL_call     c pass
>   14 VCL_return   c pass
>   14 Backend      c 27 localhost localhost
>   27 TxRequest    b GET
>   27 TxURL        b /some/uri/goes/here
>   27 TxProtocol   b HTTP/1.1
>   27 TxHeader     b Accept: image/gif, image/x-xbitmap, image/jpeg,
> image/pjpeg, application/vnd.ms-excel, application/vnd.ms-powerpoint,
> application/msword,
>  application/x-shockwave-flash, application/xaml+xml,
> application/vnd.ms-xpsdocument, application/x-ms-xbap, applicati
>   27 TxHeader     b Accept-Language: en-gb
>   27 TxHeader     b UA-CPU: x86
>   27 TxHeader     b User-Agent: Mozilla/4.0 (compatible; MSIE 7.0;
> Windows NT 5.1; .NET CLR 1.1.4322; .NET CLR 2.0.50727; .NET CLR
> 3.0.04506.30; .NET CLR 3.0
> .04506.648; .NET CLR 3.5.21022; InfoPath.2; .NET CLR 3.0.4506.2152;
> .NET CLR 3.5.30729)
>   27 TxHeader     b Host: intranet.company.com
>   27 TxHeader     b Cookie: amlbcookie=01
>   27 TxHeader     b Accept-Encoding: gzip
>   27 TxHeader     b X-Varnish: 1053467939
>   27 TxHeader     b X-Forwarded-For: 10.99.1.15
>   27 BackendClose b localhost
>   14 VCL_call     c error
>   14 VCL_return   c deliver
>   14 Length       c 466
>   14 VCL_call     c deliver
>   14 VCL_return   c deliver
>   14 TxProtocol   c HTTP/1.1
>   14 TxStatus     c 503
>   14 TxResponse   c Service Unavailable
>   14 TxHeader     c Server: Varnish
>   14 TxHeader     c Retry-After: 0
>   14 TxHeader     c Content-Type: text/html; charset=utf-8
>   14 TxHeader     c Content-Length: 466
>   14 TxHeader     c Date: Thu, 30 Jul 2009 09:57:46 GMT
>   14 TxHeader     c X-Varnish: 1053467939
>   14 TxHeader     c Age: 0
>   14 TxHeader     c Via: 1.1 varnish
>   14 TxHeader     c Connection: close
>   14 ReqEnd       c 1053467939 1248947866.962788105
> 1248947866.963104010 0.000128031 0.000283957 0.000031948
>   14 SessionClose c error
>