intermittent 503's

[Geoffrey Simmons]

>    35 FetchError   c backend write error: 104 (Connection reset by peer)

"backend write error" in versions up to at least 3.0.3 can be a very misleading error message -- you may in fact have had a connection reset on the *client* connection while reading the request body of your POST request. In my experience, that's much more common than an error writing to the backend connection.

Sending the request body to the backend is part of the fetch operation, which is why an error in that phase is logged as a FetchError. Varnish reads the body from the client and writes it to the backend in a loop, so if any error occurs then, either on the client or backend side, it's logged as "FetchError: backend write error".

There was a fix that made the error message distinguish the problem more clearly, might have made it into 3.0.4.

Anyway if I were you I'd look for a connection reset on a read of the POST body from the client connection.


HTH,
Geoff

Sent from my iPad

> On Apr 17, 2014, at 12:37 AM, L Cruzero <lcruzero at gmail.com> wrote:
> 
> Hi,  in our varnish setup(varnish-3.0.3-1.el5) using a loadbalanced really Healthy VIP as the only backend, we seem to be getting intermittent 503's 
> 
> 
> stdout captured with varnishlog -c -m TxStatus:503 
> 
>    35 ReqStart     c 10.33.13.254 24582 1166630723
>    35 RxRequest    c POST
>    35 RxURL        c /index.php?option=com_jce&task=plugin&plugin=imgmanager&file=imgmanager&method=form&cid=20&6bc427c8a7981f4fe1f5ac65c1246b5f=cf6dd3cf1923c950586d0dd595c8e20b
>    35 RxProtocol   c HTTP/1.1
>    35 RxHeader     c User-Agent: BOT/0.1 (BOT for JCE)
>    35 RxHeader     c Content-Type: multipart/form-data; boundary=---------------------------41184676334
>    35 RxHeader     c Content-Length: 5000
>    35 RxHeader     c True-Client-IP: 202.80.119.178
>    35 RxHeader     c X-Akamai-CONFIG-LOG-DETAIL: true
>    35 RxHeader     c TE:  chunked;q=1.0
>    35 RxHeader     c Connection: TE
>    35 RxHeader     c Akamai-Origin-Hop: 2
>    35 RxHeader     c Via: 1.1 v1-akamaitech.net(ghost) (AkamaiGHost), 1.1 akamai.net(ghost) (AkamaiGHost)
>    35 RxHeader     c X-Forwarded-For: 202.80.119.178, 114.4.39.206
>    35 RxHeader     c Host: www.somedomain.com
>    35 RxHeader     c Cache-Control: max-age=120
>    35 RxHeader     c Connection: keep-alive
>    35 RxHeader     c X-Forwarded-For: 23.73.180.223
>    35 RxHeader     c Accept-Encoding: identity
>    35 VCL_call     c recv pass
>    35 VCL_call     c hash
>    35 Hash         c /index.php?option=com_jce&task=plugin&plugin=imgmanager&file=imgmanager&method=form&cid=20&6bc427c8a7981f4fe1f5ac65c1246b5f=cf6dd3cf1923c950586d0dd595c8e20b
>    35 Hash         c www.somedomain.com
>    35 VCL_return   c hash
>    35 VCL_call     c pass pass
>    35 Backend      c 29 www_prod www_prod
>    35 FetchError   c backend write error: 104 (Connection reset by peer)
>    35 Backend      c 42 www_prod www_prod
>    35 FetchError   c backend write error: 104 (Connection reset by peer)
>    35 VCL_call     c error deliver
>    35 VCL_call     c deliver deliver
>    35 TxProtocol   c HTTP/1.1
>    35 TxStatus     c 503
>    35 TxResponse   c Service Unavailable
>    35 TxHeader     c Server: Varnish
>    35 TxHeader     c Content-Type: text/html; charset=utf-8
>    35 TxHeader     c Retry-After: 5
>    35 TxHeader     c Content-Length: 419
>    35 TxHeader     c Accept-Ranges: bytes
>    35 TxHeader     c Date: Tue, 15 Apr 2014 19:31:22 GMT
>    35 TxHeader     c X-Varnish: 1166630723
>    35 TxHeader     c Age: 0
>    35 TxHeader     c Via: 1.1 varnish
>    35 TxHeader     c Connection: close
>    35 Length       c 419
>    35 ReqEnd       c 1166630723 1397590282.467632055 1397590282.899372101 0.033779144 0.431685925 0.0000541 
> 
> 
> here is super simple pass thru NO cache config :
> 
> 
> backend www_prod {
>   .host = "cs-****.*****.*****.net";
>   .port = "80";
>   .probe = {
>         .url = "/";
>         .timeout = 5s;
>         .interval = 1s;
>         .window = 5;
>         .threshold = 2;
>   }
> }
> 
>  sub vcl_recv {
>       if (req.http.X-ADI-MISS) {
>         # Force a cache miss
>                  set req.hash_always_miss = true;
>         }
> 
>         if (req.url == "/varnish-health/" || req.url ~ "^/stack-check*") {
>                 error 200 "Varnish is responding";
>                 set req.http.Connection = "close";
>   }
> 
> 
>     if (req.http.host ~ "^(origin|www)") {
>         set req.http.host = regsub(req.http.host, "^origin\.", "www.");
>         set req.backend = www_prod;
>         return(pass);
> 
> }
> 
> 
> ## we tried with and w/o these conditions intermittent 503's persisted 
> 
> #       if (req.backend.healthy) {
> #       set req.grace = 30s;
> #   } else {
> #       set req.grace = 24h;
> #   }
> 
>  }
> 
> 
> sub vcl_fetch {
> #            set beresp.do_esi = true;
>             set beresp.ttl = 1m;
> 
> #           set beresp.grace = 24h;
> }
> 
> 
> Any helpful thoughts and or possible leads on this will be much appreciated.
> kind regards,
> -LC
> 
> _______________________________________________
> varnish-misc mailing list
> varnish-misc at varnish-cache.org
> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/varnish-misc/attachments/20140417/cf00e773/attachment.html>