<html><head><meta http-equiv="content-type" content="text/html; charset=utf-8"></head><body dir="auto"><div><blockquote type="cite"><div dir="ltr"> 35 FetchError c backend write error: 104 (Connection reset by peer)</div></blockquote><div><br></div>"backend write error" in versions up to at least 3.0.3 can be a very misleading error message -- you may in fact have had a connection reset on the *client* connection while reading the request body of your POST request. In my experience, that's much more common than an error writing to the backend connection.</div><div><br></div><div>Sending the request body to the backend is part of the fetch operation, which is why an error in that phase is logged as a FetchError. Varnish reads the body from the client and writes it to the backend in a loop, so if any error occurs then, either on the client or backend side, it's logged as "FetchError: backend write error".</div><div><br></div><div>There was a fix that made the error message distinguish the problem more clearly, might have made it into 3.0.4.</div><div><br></div><div>Anyway if I were you I'd look for a connection reset on a read of the POST body from the client connection.</div><div><br></div><div><br></div><div>HTH,</div><div>Geoff</div><div><br>Sent from my iPad</div><div><br>On Apr 17, 2014, at 12:37 AM, L Cruzero <<a href="mailto:lcruzero@gmail.com">lcruzero@gmail.com</a>> wrote:<br><br></div><blockquote type="cite"><div><div dir="ltr">Hi, in our varnish setup(varnish-3.0.3-1.el5) using a loadbalanced really Healthy VIP as the only backend, we seem to be getting intermittent 503's <div><br></div><div><br></div><div>stdout captured with varnishlog -c -m TxStatus:503 </div>
<div><br></div><div> 35 ReqStart c 10.33.13.254 24582 1166630723</div><div> 35 RxRequest c POST</div>
<div> 35 RxURL c /index.php?option=com_jce&task=plugin&plugin=imgmanager&file=imgmanager&method=form&cid=20&6bc427c8a7981f4fe1f5ac65c1246b5f=cf6dd3cf1923c950586d0dd595c8e20b</div><div> 35 RxProtocol c HTTP/1.1</div>
<div> 35 RxHeader c User-Agent: BOT/0.1 (BOT for JCE)</div><div> 35 RxHeader c Content-Type: multipart/form-data; boundary=---------------------------41184676334</div><div> 35 RxHeader c Content-Length: 5000</div>
<div> 35 RxHeader c True-Client-IP: 202.80.119.178</div><div> 35 RxHeader c X-Akamai-CONFIG-LOG-DETAIL: true</div><div> 35 RxHeader c TE: chunked;q=1.0</div><div> 35 RxHeader c Connection: TE</div>
<div> 35 RxHeader c Akamai-Origin-Hop: 2</div><div> 35 RxHeader c Via: 1.1 <a href="http://v1-akamaitech.net" target="_blank">v1-akamaitech.net</a>(ghost) (AkamaiGHost), 1.1 <a href="http://akamai.net" target="_blank">akamai.net</a>(ghost) (AkamaiGHost)</div>
<div> 35 RxHeader c X-Forwarded-For: 202.80.119.178, 114.4.39.206</div><div> 35 RxHeader c Host: <a href="http://www.oregonlive.com" target="_blank">www.somedomain.com</a></div><div> 35 RxHeader c Cache-Control: max-age=120</div>
<div> 35 RxHeader c Connection: keep-alive</div><div> 35 RxHeader c X-Forwarded-For: 23.73.180.223</div><div> 35 RxHeader c Accept-Encoding: identity</div><div> 35 VCL_call c recv pass</div><div> 35 VCL_call c hash</div>
<div> 35 Hash c /index.php?option=com_jce&task=plugin&plugin=imgmanager&file=imgmanager&method=form&cid=20&6bc427c8a7981f4fe1f5ac65c1246b5f=cf6dd3cf1923c950586d0dd595c8e20b</div><div> 35 Hash c <a href="http://www.oregonlive.com" target="_blank">www.somedomain.com</a></div>
<div> 35 VCL_return c hash</div><div> 35 VCL_call c pass pass</div><div> 35 Backend c 29 www_prod www_prod</div><div> 35 FetchError c backend write error: 104 (Connection reset by peer)</div><div> 35 Backend c 42 www_prod www_prod</div>
<div> 35 FetchError c backend write error: 104 (Connection reset by peer)</div><div> 35 VCL_call c error deliver</div><div> 35 VCL_call c deliver deliver</div><div> 35 TxProtocol c HTTP/1.1</div><div>
35 TxStatus c 503</div>
<div> 35 TxResponse c Service Unavailable</div><div> 35 TxHeader c Server: Varnish</div><div> 35 TxHeader c Content-Type: text/html; charset=utf-8</div><div> 35 TxHeader c Retry-After: 5</div><div> 35 TxHeader c Content-Length: 419</div>
<div> 35 TxHeader c Accept-Ranges: bytes</div><div> 35 TxHeader c Date: Tue, 15 Apr 2014 19:31:22 GMT</div><div> 35 TxHeader c X-Varnish: 1166630723</div><div> 35 TxHeader c Age: 0</div><div> 35 TxHeader c Via: 1.1 varnish</div>
<div> 35 TxHeader c Connection: close</div><div> 35 Length c 419</div><div> 35 ReqEnd c 1166630723 1397590282.467632055 1397590282.899372101 0.033779144 0.<a href="tel:431685925%200" value="+14316859250" target="_blank">431685925 0</a>.0000541 </div>
<div><br></div><div><br></div><div>here is super simple pass thru NO cache config :</div><div><br></div><div><br></div><div><div>backend www_prod {</div><div> .host = "cs-****.*****.*****.net";</div><div> .port = "80";</div>
<div> .probe = {</div><div> .url = "/";</div><div> .timeout = 5s;</div><div> .interval = 1s;</div><div> .window = 5;</div><div> .threshold = 2;</div><div> }</div><div>}</div>
<div><br></div><div> sub vcl_recv {</div><div> if (req.http.X-ADI-MISS) {</div><div> # Force a cache miss</div><div> set req.hash_always_miss = true;</div><div> }</div><div><br></div><div>
if (req.url == "/varnish-health/" || req.url ~ "^/stack-check*") {</div><div> error 200 "Varnish is responding";</div><div> set req.http.Connection = "close";</div>
<div> }</div><div><br></div><div><br></div><div> if (req.http.host ~ "^(origin|www)") {</div><div> set req.http.host = regsub(req.http.host, "^origin\.", "www.");</div><div> set req.backend = www_prod;</div>
<div> return(pass);</div><div><br></div><div>}</div><div><br></div><div><br></div><div>## we tried with and w/o these conditions intermittent 503's persisted </div><div><br></div><div># if (req.backend.healthy) {</div>
<div># set req.grace = 30s;</div><div># } else {<br></div><div># set req.grace = 24h;</div><div># }</div><div><br></div><div> }</div><div><br></div><div><br></div><div>sub vcl_fetch {</div><div># set beresp.do_esi = true;</div>
<div> set beresp.ttl = 1m;</div><div><br></div><div># set beresp.grace = 24h;</div><div>}</div></div><div><br></div><div><br></div><div><pre style="margin-top:1.5em;margin-bottom:1.5em;padding:0px;border:0px;font-family:'andale mono','lucida console',monospace;font-size:12px;line-height:18.001798629760742px;vertical-align:baseline">Any helpful thoughts and or possible leads on this will be much appreciated.</pre><pre style="margin-top:1.5em;margin-bottom:1.5em;padding:0px;border:0px;font-family:'andale mono','lucida console',monospace;font-size:12px;line-height:18.001798629760742px;vertical-align:baseline">kind regards,</pre><pre style="margin-top:1.5em;margin-bottom:1.5em;padding:0px;border:0px;font-family:'andale mono','lucida console',monospace;font-size:12px;line-height:18.001798629760742px;vertical-align:baseline">-LC
</pre></div></div>
</div></blockquote><blockquote type="cite"><div><span>_______________________________________________</span><br><span>varnish-misc mailing list</span><br><span><a href="mailto:varnish-misc@varnish-cache.org">varnish-misc@varnish-cache.org</a></span><br><span><a href="https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc">https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc</a></span></div></blockquote></body></html>