Sporadic "backend write error: 11" errors

[HC Saustrup]

  Hi everyone!

I've recently deployed Varnish 2.1.3 on a 64-bit CentOS 5.5, but it 
seems to fail on certain requests under stress. It's a POST request 
that's PASSed through to the backend. The backend is rather bulky 
Windows CMS, but it usually answers these requests quite fast.

According to the logs, it fails while talking to the backend and returns 
a 503 error to the client after hanging for approximately 5000ms. The 
Windows backend has a connection timeout of 120s, and Varnish is 
configured with a timeout of 60s (connect_timeout, first_byte_timeout 
and between_bytes_timeout) so that's a bit weird.

This happens in approximately 5-10% of these requests, which doesn't 
point to bad mojo in my VCL files - but I could be wrong. Let me know if 
these would be helpful.

Here's a typical failed request:

    46 Debug        c "herding"
    46 SessionClose c timeout
    46 StatSess     c x.x.x.x 16838 7 1 3 0 2 2 833 52886
    46 SessionOpen  c x.x.x.x 20290 :80
    46 Debug        c "herding"
    46 ReqStart     c x.x.x.x 20290 2127593665
    46 RxRequest    c POST
    46 RxURL        c /webservices/RelatedProducts.asmx/GetRelatedProducts
    46 RxProtocol   c HTTP/1.1
    46 RxHeader     c Accept: */*
    46 RxHeader     c Accept-Language: da
    46 RxHeader     c Referer: http://xxx
    46 RxHeader     c Content-Type: application/json; charset=utf-8
    46 RxHeader     c UA-CPU: x86
    46 RxHeader     c Accept-Encoding: gzip, deflate
    46 RxHeader     c User-Agent: Mozilla/4.0 (compatible; MSIE 7.0; 
Windows NT 6.0; SLCC1; .NET CLR 2.0.50727; Media Center PC 5.0; .NET CLR 
3.5.30729; .NET CLR 3.0.30618)
    46 RxHeader     c Host: xxx
    46 RxHeader     c Content-Length: 106
    46 RxHeader     c Connection: Keep-Alive
    46 RxHeader     c Cache-Control: no-cache
    46 RxHeader     c Cookie: xxx
    46 VCL_call     c recv
    46 VCL_acl      c MATCH allowedClients 0.0.0.0/0
    46 VCL_return   c pass
    46 VCL_call     c hash
    46 VCL_return   c hash
    46 VCL_call     c pass
    46 VCL_return   c pass
    46 Backend      c 114 default default
    46 FetchError   c backend write error: 11 <-- oops!
    46 VCL_call     c error
    46 VCL_return   c deliver
    46 VCL_call     c deliver
    46 VCL_return   c deliver
    46 TxProtocol   c HTTP/1.1
    46 TxStatus     c 503
    46 TxResponse   c Service Unavailable
    46 TxHeader     c Server: Varnish
    46 TxHeader     c Retry-After: 0
    46 TxHeader     c Content-Type: text/html; charset=utf-8
    46 TxHeader     c Content-Length: 1012
    46 TxHeader     c Date: Fri, 30 Jul 2010 16:22:01 GMT
    46 TxHeader     c Age: 5
    46 TxHeader     c Connection: close
    46 TxHeader     c X-Cache: MISS
    46 Length       c 1012
    46 ReqEnd       c 2127593665 1280506916.547359943 
1280506921.548347950 0.000546932 5.000926018 0.000061989

Let me know if there's anything I can do to help resolve this.

Cheers,
HC

-- 
Hans Christian Saustrup
Partner, CodeCompany.dk, Skolebakken 7, 8000 Århus C
Mobil +45-30261024