<html><body bgcolor="#FFFFFF"><div><br></div><div><br>On 8 Mar 2011, at 16:07, Ben Dodd <<a href="mailto:B.Dodd@comicrelief.com">B.Dodd@comicrelief.com</a>> wrote:</div><div><br></div><div></div><blockquote type="cite">
<div>
Hello,
<div><br>
<div>This is only to add we've been experiencing exactly the same issue and are desperately searching for a solution. Can anyone help?</div>
<div><br>
</div>
<div>Thanks, Ben</div>
<div><br>
<div>
<div>On 8 Mar 2011, at 21:55, <<a href="mailto:varnish-misc-request@varnish-cache.org"><a href="mailto:varnish-misc-request@varnish-cache.org">varnish-misc-request@varnish-cache.org</a></a>> <<a href="mailto:varnish-misc-request@varnish-cache.org"><a href="mailto:varnish-misc-request@varnish-cache.org">varnish-misc-request@varnish-cache.org</a></a>> wrote:</div>
<br class="Apple-interchange-newline">
<blockquote><span class="Apple-style-span" style="border-collapse: separate; font-family: Helvetica; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: 2; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-border-horizontal-spacing: 0px; -webkit-border-vertical-spacing: 0px; -webkit-text-decorations-in-effect: none; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; font-size: medium; ">-----Original
Message-----<br>
From:<span class="Apple-converted-space"> </span><a href="mailto:varnish-misc-bounces@varnish-cache.org"><a href="mailto:varnish-misc-bounces@varnish-cache.org">varnish-misc-bounces@varnish-cache.org</a></a><br>
[mailto:<a href="mailto:varnish-misc-bounces@varnish-cache.org">varnish-misc-bounces@varnish-cache.org</a>] On Behalf Of Ronan<br>
Mullally<br>
Sent: March-08-11 3:38 PM<br>
To:<span class="Apple-converted-space"> </span><a href="mailto:varnish-misc@varnish-cache.org"><a href="mailto:varnish-misc@varnish-cache.org">varnish-misc@varnish-cache.org</a></a><br>
Subject: Varnish 503ing on ~1/100 POSTs<br>
<br>
I'm seeing intermittant 503s on POSTs to a fairly busy VBulletin<br>
website.<br>
The current load is light (up to a couple of thousand active sessions,<br>
peak is around five thousand). Varnish has a fairly simple config with<br>
a director consisting of two Apache backends:<br>
<br>
backend backend1 {<br>
.host = "1.2.3.4";<br>
.port = "80";<br>
.connect_timeout = 5s;<br>
.first_byte_timeout = 90s;<br>
.between_bytes_timeout = 90s;<br>
.probe = {<br>
.timeout = 5s;<br>
.interval = 5s;<br>
.window = 5;<br>
.threshold = 3;<br>
.request =<br>
"HEAD /favicon.ico HTTP/1.0"<br>
"X-Forwarded-For: 1.2.3.4"<br>
"Connection: close";<br>
}<br>
}<br>
<br>
backend backend2 {<br>
.host = "5.6.7.8";<br>
.port = "80";<br>
.connect_timeout = 5s;<br>
.first_byte_timeout = 90s;<br>
.between_bytes_timeout = 90s;<br>
.probe = {<br>
.timeout = 5s;<br>
.interval = 5s;<br>
.window = 5;<br>
.threshold = 3;<br>
.request =<br>
"HEAD /favicon.ico HTTP/1.0"<br>
"X-Forwarded-For: 5.6.7.8"<br>
"Connection: close";<br>
}<br>
}<br>
<br>
The numbers are modest, but significant - about 1 POST in a hundred<br>
fails.<br>
I've upped the backend timeouts to 90 seconds (first_byte /<br>
between_bytes)<br>
and I'm pretty confident they're responding in well under that time.<br>
<br>
varnishlog does not show any backend health changes. A typical event<br>
looks like:<br>
<br>
Varnish:<br>
a.b.c.d - - [08/Mar/2011:14:48:03 +0000] "POST<br>
<a href="http://www.sitename.net/newreply.php?do=postreply&t=285227"><a href="http://www.sitename.net/newreply.php?do=postreply&t=285227">http://www.sitename.net/newreply.php?do=postreply&t=285227</a></a><span class="Apple-converted-space"> </span>HTTP/1.1" 503<br>
2623<br>
<br>
Backend:<br>
a.b.c.d - - [08/Mar/2011:14:48:03 +0000] "POST<br>
/newreply.php?do=postreply&t=285227 HTTP/1.1" 200 2686<br>
<br>
The POST appears to work fine on the backend but the user gets a 503<br>
from<br>
Varnish. It's not unusual to see users getting the error several times<br>
in<br>
a row (presumably re-submitting the post):<br>
<br>
a.b.c.d - - [08/Mar/2011:18:21:23 +0000] "POST<br>
<a href="http://www.sitename.net/editpost.php?do=updatepost&p=9405408"><a href="http://www.sitename.net/editpost.php?do=updatepost&p=9405408">http://www.sitename.net/editpost.php?do=updatepost&p=9405408</a></a><span class="Apple-converted-space"> </span>HTTP/1.1"<br>
503 2623<br>
a.b.c.d - - [08/Mar/2011:18:21:36 +0000] "POST<br>
<a href="http://www.sitename.net/editpost.php?do=updatepost&p=9405408"><a href="http://www.sitename.net/editpost.php?do=updatepost&p=9405408">http://www.sitename.net/editpost.php?do=updatepost&p=9405408</a></a><span class="Apple-converted-space"> </span>HTTP/1.1"<br>
503 2623<br>
a.b.c.d - - [08/Mar/2011:18:21:50 +0000] "POST<br>
<a href="http://www.sitename.net/editpost.php?do=updatepost&p=9405408"><a href="http://www.sitename.net/editpost.php?do=updatepost&p=9405408">http://www.sitename.net/editpost.php?do=updatepost&p=9405408</a></a><span class="Apple-converted-space"> </span>HTTP/1.1"<br>
503 2623<br>
<br>
A typical request is below. The first attempt fails with:<br>
<br>
33 FetchError c http first read error: -1 0 (Success)<br>
<br>
there is presumably a restart and the second attempt (sometimes to<br>
backend1, sometimes backend2) fails with:<br>
<br>
33 FetchError c backend write error: 11 (Resource temporarily<br>
unavailable)<br>
<br>
This pattern has been the same on the few transactions I've examined in<br>
detail. The full log output of a typical request is below.<br>
<br>
I'm stumped. Has anybody got any ideas what might be causing this?<br>
<br>
<br>
-Ronan<br>
<br>
<br>
33 RxRequest c POST<br>
33 RxURL c /ajax.php<br>
33 RxProtocol c HTTP/1.1<br>
33 RxHeader c Accept: */*<br>
33 RxHeader c Accept-Language: nl-be<br>
33 RxHeader c Referer:<span class="Apple-converted-space"> </span><a href="http://www.redcafe.net/"><a href="http://www.redcafe.net/">http://www.redcafe.net/</a></a><br>
33 RxHeader c x-requested-with: XMLHttpRequest<br>
33 RxHeader c Content-Type: application/x-www-form-urlencoded;<br>
charset=UTF-8<br>
33 RxHeader c Accept-Encoding: gzip, deflate<br>
33 RxHeader c User-Agent: Mozilla/4.0 (compatible; ...)<br>
33 RxHeader c Host:<span class="Apple-converted-space"> </span><a href="http://www.sitename.net/"><a href="http://www.sitename.net">www.sitename.net</a></a><br>
33 RxHeader c Content-Length: 82<br>
33 RxHeader c Connection: Keep-Alive<br>
33 RxHeader c Cache-Control: no-cache<br>
33 RxHeader c Cookie: ...<br>
33 VCL_call c recv<br>
33 VCL_return c pass<br>
33 VCL_call c hash<br>
33 VCL_return c hash<br>
33 VCL_call c pass<br>
33 VCL_return c pass<br>
33 Backend c 44 backend backend1<br>
44 TxRequest b POST<br>
44 TxURL b /ajax.php<br>
44 TxProtocol b HTTP/1.1<br>
44 TxHeader b Accept: */*<br>
44 TxHeader b Accept-Language: nl-be<br>
44 TxHeader b Referer:<span class="Apple-converted-space"> </span><a href="http://www.sitename.net/"><a href="http://www.sitename.net/">http://www.sitename.net/</a></a><br>
44 TxHeader b x-requested-with: XMLHttpRequest<br>
44 TxHeader b Content-Type: application/x-www-form-urlencoded;<br>
charset=UTF-8<br>
44 TxHeader b User-Agent: Mozilla/4.0 (compatible; ...)<br>
44 TxHeader b Host:<span class="Apple-converted-space"> </span><a href="http://www.sitename.net/"><a href="http://www.sitename.net">www.sitename.net</a></a><br>
44 TxHeader b Content-Length: 82<br>
44 TxHeader b Cache-Control: no-cache<br>
44 TxHeader b Cookie: ...<br>
44 TxHeader b Accept-Encoding: gzip<br>
44 TxHeader b X-Forwarded-For: a.b.c.d<br>
44 TxHeader b X-Varnish: 657185708<br>
* 33 FetchError c http first read error: -1 0 (Success)<br>
44 BackendClose b backend1<br>
33 Backend c 47 backend backend2<br>
47 TxRequest b POST<br>
47 TxURL b /ajax.php<br>
47 TxProtocol b HTTP/1.1<br>
47 TxHeader b Accept: */*<br>
47 TxHeader b Accept-Language: nl-be<br>
47 TxHeader b Referer:<span class="Apple-converted-space"> </span><a href="http://www.sitename.net/"><a href="http://www.sitename.net/">http://www.sitename.net/</a></a><br>
47 TxHeader b x-requested-with: XMLHttpRequest<br>
47 TxHeader b Content-Type: application/x-www-form-urlencoded;<br>
charset=UTF-8<br>
47 TxHeader b User-Agent: Mozilla/4.0 (compatible; ...)<br>
47 TxHeader b Host:<span class="Apple-converted-space"> </span><a href="http://www.sitename.net/"><a href="http://www.sitename.net">www.sitename.net</a></a><br>
47 TxHeader b Content-Length: 82<br>
47 TxHeader b Cache-Control: no-cache<br>
47 TxHeader b Cookie: ...<br>
47 TxHeader b Accept-Encoding: gzip<br>
47 TxHeader b X-Forwarded-For: a.b.c.d<br>
47 TxHeader b X-Varnish: 657185708<br>
* 33 FetchError c backend write error: 11 (Resource temporarily<br>
unavailable)<br>
47 BackendClose b backend2<br>
33 VCL_call c error<br>
33 VCL_return c deliver<br>
33 VCL_call c deliver<br>
33 VCL_return c deliver<br>
33 TxProtocol c HTTP/1.1<br>
33 TxStatus c 503<br>
33 TxResponse c Service Unavailable<br>
33 TxHeader c Server: Varnish<br>
33 TxHeader c Retry-After: 0<br>
33 TxHeader c Content-Type: text/html; charset=utf-8<br>
33 TxHeader c Content-Length: 2623<br>
33 TxHeader c Date: Tue, 08 Mar 2011 17:08:33 GMT<br>
33 TxHeader c X-Varnish: 657185708<br>
33 TxHeader c Age: 3<br>
33 TxHeader c Via: 1.1 varnish<br>
33 TxHeader c Connection: close<br>
33 Length c 2623<br>
33 ReqEnd c 657185708 1299604110.559967279 1299604113.447372913<br>
0.000037670 2.887368441 0.000037193<br>
33 SessionClose c error<br>
33 StatSess c a.b.c.d 50044 3 1 1 0 1 0 235 2623</span></blockquote>
</div>
<br>
</div>
</div>
<br>
<hr>
<font face="Arial" color="Black" size="1">Comic Relief 1st Floor 89 Albert Embankment London SE1 7TP Tel: 020 7820 2000 Fax: 020 7820 2222 <a href="mailto:red@comicrelief.com">red@comicrelief.com</a> <a href="http://www.comicrelief.com"><a href="http://www.comicrelief.com">www.comicrelief.com</a></a><br>
<br>
Comic Relief is the operating name of Charity Projects, a company limited by guarantee and registered in England no. 1806414; registered charity 326568 (England & Wales) and SC039730 (Scotland). Comic Relief Ltd is a subsidiary of Charity Projects and registered
in England no. 1967154. Registered offices: Hanover House, 14 Hanover Square, London W1S 1HP. VAT no. 773865187.<br>
<br>
This email (and any attachment) may contain confidential and/or privileged information. If you are not the intended addressee, you must not use, disclose, copy or rely on anything in this email and should contact the sender and delete it immediately. The views
of the author are not necessarily those of Comic Relief. We cannot guarantee that this email (and any attachment) is virus free or has not been intercepted and amended, so do not accept liability for any damage resulting from software viruses. You should carry
out your own virus checks.<br>
</font>
</div></blockquote><blockquote type="cite"><div><span>_______________________________________________</span><br><span>varnish-misc mailing list</span><br><span><a href="mailto:varnish-misc@varnish-cache.org">varnish-misc@varnish-cache.org</a></span><br>
<span><a href="http://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc">http://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc</a></span></div></blockquote><br><div>Whilst this may not be a fix to a possible bug in varnish have you tried switching posts to pipe instead of pass?</div>
<div><br></div><div>Stew</div></body></html>