varnish-misc Digest, Vol 97, Issue 16
L Cruzero
lcruzero at gmail.com
Thu Apr 17 13:51:48 CEST 2014
Geoff, Et al, thanks for your replies, i should have mentioned, that the
503's are not unique to POST requests, they are also happening more
frequent within
(very rarely == on 0.0038% of requests), on a GETs. i've been thinking of
testing within my config something along this FOUND vcl code snipped as a
possible solution.
*sub vcl_fetch {
** if (obj.status == 500 || obj.status == 503 || obj.status ==
** 504) {
** restart;
** }*
174 ReqStart c 10.33.13.254 38233 1505576793
174 RxRequest c GET
174 RxURL c /auburnfootball/modules/most_read.html
174 RxProtocol c HTTP/1.1
174 RxHeader c If-Modified-Since: Wed, 16 Apr 2014 14:23:34 GMT
174 RxHeader c Accept:
text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
174 RxHeader c Accept-Charset: utf-8, iso-8859-1, utf-16, *;q=0.7
174 RxHeader c Accept-Language: en-US
174 RxHeader c Cookie: adv_lid=04ea3e9b86834130c4cb3be359f7b220;
OAX=RsRvNFEDhgkAAuO2; __qca=P0-1959933173-1359185426964;
__atuvc=0%7C27%2C0%7C28%2C0%7C29%2C0%7C30%2C1%7C31;
mp_948b359d7741fa246376d95399585cc3_mixpanel=%7B%7D;
__utma=130511327.2000905843.1375913741.1375
174 RxHeader c User-Agent: Mozilla/5.0 (Linux; U; Android 4.1.2;
en-us; SCH-I200 Build/JZO54K) AppleWebKit/534.30 (KHTML, like Gecko)
Version/4.0 Mobile Safari/534.30
174 RxHeader c X-UIDH:
MTQ5ODI0NDA2MAA/58mlyaG6h8c9ATsT9BtivrzyjC/Gl0mdhNCPRIOC1Q==
174 RxHeader c x-wap-profile:
http://uaprof.vtext.com/sam/SCH-I200/SCH-I200.xml
174 RxHeader c X-Akamai-Esi-Fragment-Suffix: f.6
174 RxHeader c True-Client-IP: 127.0.0.1
174 RxHeader c X-Akamai-CONFIG-LOG-DETAIL: true
174 RxHeader c TE: chunked;q=1.0
174 RxHeader c Connection: TE
174 RxHeader c Akamai-Origin-Hop: 2
174 RxHeader c Via: 4.0 asi_server, 1.0 v1-akamaitech.net(ghost)
(AkamaiGHost), 1.1 akamai.net(ghost) (AkamaiGHost)
174 RxHeader c Accept-ESI: 1.0
174 RxHeader c X-Forwarded-For: 127.0.0.1, 96.17.202.197
174 RxHeader c Host: www.al.com
174 RxHeader c Cache-Control: max-age=180
174 RxHeader c Connection: keep-alive
174 RxHeader c X-Forwarded-For: 23.73.180.223
174 RxHeader c Accept-Encoding: identity
174 VCL_call c recv pass
174 VCL_call c hash
174 Hash c /auburnfootball/modules/most_read.html
174 Hash c www.**.com
174 VCL_return c hash
174 VCL_call c pass pass
174 Backend c 101 www_prod www_prod
174 FetchError c http first read error: -1 104 (Connection reset by
peer)
174 Backend c 56 www_prod www_prod
174 FetchError c http first read error: -1 104 (Connection reset by
peer)
174 VCL_call c error deliver
174 VCL_call c deliver deliver
174 TxProtocol c HTTP/1.1
174 TxStatus c 503
174 TxResponse c Service Unavailable
174 TxHeader c Server: Varnish
174 TxHeader c Content-Type: text/html; charset=utf-8
174 TxHeader c Retry-After: 5
174 TxHeader c Content-Length: 419
174 TxHeader c Accept-Ranges: bytes
174 TxHeader c Date: Wed, 16 Apr 2014 16:30:29 GMT
174 TxHeader c X-Varnish: 1505576793
174 TxHeader c Age: 0
174 TxHeader c Via: 1.1 varnish
174 TxHeader c Connection: close
174 Length c 419
174 ReqEnd c 1505576793 1397665829.285193920 1397665829.298522949
0.022354841 0.013305187 0.000023842
On Thu, Apr 17, 2014 at 6:00 AM, <varnish-misc-request at varnish-cache.org>wrote:
> Send varnish-misc mailing list submissions to
> varnish-misc at varnish-cache.org
>
> To subscribe or unsubscribe via the World Wide Web, visit
> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
> or, via email, send a message with subject or body 'help' to
> varnish-misc-request at varnish-cache.org
>
> You can reach the person managing the list at
> varnish-misc-owner at varnish-cache.org
>
> When replying, please edit your Subject line so it is more specific
> than "Re: Contents of varnish-misc digest..."
>
>
> Today's Topics:
>
> 1. Re: intermittent 503's (Geoffrey Simmons)
>
>
> ----------------------------------------------------------------------
>
> Message: 1
> Date: Thu, 17 Apr 2014 17:46:21 +1200
> From: Geoffrey Simmons <geoff at uplex.de>
> To: L Cruzero <lcruzero at gmail.com>
> Cc: "varnish-misc at varnish-cache.org" <varnish-misc at varnish-cache.org>
> Subject: Re: intermittent 503's
> Message-ID: <A1B813DE-4217-4546-9E65-E0D9139C684B at uplex.de>
> Content-Type: text/plain; charset="us-ascii"
>
> > 35 FetchError c backend write error: 104 (Connection reset by peer)
>
> "backend write error" in versions up to at least 3.0.3 can be a very
> misleading error message -- you may in fact have had a connection reset on
> the *client* connection while reading the request body of your POST
> request. In my experience, that's much more common than an error writing to
> the backend connection.
>
> Sending the request body to the backend is part of the fetch operation,
> which is why an error in that phase is logged as a FetchError. Varnish
> reads the body from the client and writes it to the backend in a loop, so
> if any error occurs then, either on the client or backend side, it's logged
> as "FetchError: backend write error".
>
> There was a fix that made the error message distinguish the problem more
> clearly, might have made it into 3.0.4.
>
> Anyway if I were you I'd look for a connection reset on a read of the POST
> body from the client connection.
>
>
> HTH,
> Geoff
>
> Sent from my iPad
>
> > On Apr 17, 2014, at 12:37 AM, L Cruzero <lcruzero at gmail.com> wrote:
> >
> > Hi, in our varnish setup(varnish-3.0.3-1.el5) using a loadbalanced
> really Healthy VIP as the only backend, we seem to be getting intermittent
> 503's
> >
> >
> > stdout captured with varnishlog -c -m TxStatus:503
> >
> > 35 ReqStart c 10.33.13.254 24582 1166630723
> > 35 RxRequest c POST
> > 35 RxURL c
> /index.php?option=com_jce&task=plugin&plugin=imgmanager&file=imgmanager&method=form&cid=20&6bc427c8a7981f4fe1f5ac65c1246b5f=cf6dd3cf1923c950586d0dd595c8e20b
> > 35 RxProtocol c HTTP/1.1
> > 35 RxHeader c User-Agent: BOT/0.1 (BOT for JCE)
> > 35 RxHeader c Content-Type: multipart/form-data;
> boundary=---------------------------41184676334
> > 35 RxHeader c Content-Length: 5000
> > 35 RxHeader c True-Client-IP: 202.80.119.178
> > 35 RxHeader c X-Akamai-CONFIG-LOG-DETAIL: true
> > 35 RxHeader c TE: chunked;q=1.0
> > 35 RxHeader c Connection: TE
> > 35 RxHeader c Akamai-Origin-Hop: 2
> > 35 RxHeader c Via: 1.1 v1-akamaitech.net(ghost) (AkamaiGHost),
> 1.1 akamai.net(ghost) (AkamaiGHost)
> > 35 RxHeader c X-Forwarded-For: 202.80.119.178, 114.4.39.206
> > 35 RxHeader c Host: www.somedomain.com
> > 35 RxHeader c Cache-Control: max-age=120
> > 35 RxHeader c Connection: keep-alive
> > 35 RxHeader c X-Forwarded-For: 23.73.180.223
> > 35 RxHeader c Accept-Encoding: identity
> > 35 VCL_call c recv pass
> > 35 VCL_call c hash
> > 35 Hash c
> /index.php?option=com_jce&task=plugin&plugin=imgmanager&file=imgmanager&method=form&cid=20&6bc427c8a7981f4fe1f5ac65c1246b5f=cf6dd3cf1923c950586d0dd595c8e20b
> > 35 Hash c www.somedomain.com
> > 35 VCL_return c hash
> > 35 VCL_call c pass pass
> > 35 Backend c 29 www_prod www_prod
> > 35 FetchError c backend write error: 104 (Connection reset by peer)
> > 35 Backend c 42 www_prod www_prod
> > 35 FetchError c backend write error: 104 (Connection reset by peer)
> > 35 VCL_call c error deliver
> > 35 VCL_call c deliver deliver
> > 35 TxProtocol c HTTP/1.1
> > 35 TxStatus c 503
> > 35 TxResponse c Service Unavailable
> > 35 TxHeader c Server: Varnish
> > 35 TxHeader c Content-Type: text/html; charset=utf-8
> > 35 TxHeader c Retry-After: 5
> > 35 TxHeader c Content-Length: 419
> > 35 TxHeader c Accept-Ranges: bytes
> > 35 TxHeader c Date: Tue, 15 Apr 2014 19:31:22 GMT
> > 35 TxHeader c X-Varnish: 1166630723
> > 35 TxHeader c Age: 0
> > 35 TxHeader c Via: 1.1 varnish
> > 35 TxHeader c Connection: close
> > 35 Length c 419
> > 35 ReqEnd c 1166630723 1397590282.467632055
> 1397590282.899372101 0.033779144 0.431685925 0.0000541
> >
> >
> > here is super simple pass thru NO cache config :
> >
> >
> > backend www_prod {
> > .host = "cs-****.*****.*****.net";
> > .port = "80";
> > .probe = {
> > .url = "/";
> > .timeout = 5s;
> > .interval = 1s;
> > .window = 5;
> > .threshold = 2;
> > }
> > }
> >
> > sub vcl_recv {
> > if (req.http.X-ADI-MISS) {
> > # Force a cache miss
> > set req.hash_always_miss = true;
> > }
> >
> > if (req.url == "/varnish-health/" || req.url ~ "^/stack-check*")
> {
> > error 200 "Varnish is responding";
> > set req.http.Connection = "close";
> > }
> >
> >
> > if (req.http.host ~ "^(origin|www)") {
> > set req.http.host = regsub(req.http.host, "^origin\.", "www.");
> > set req.backend = www_prod;
> > return(pass);
> >
> > }
> >
> >
> > ## we tried with and w/o these conditions intermittent 503's persisted
> >
> > # if (req.backend.healthy) {
> > # set req.grace = 30s;
> > # } else {
> > # set req.grace = 24h;
> > # }
> >
> > }
> >
> >
> > sub vcl_fetch {
> > # set beresp.do_esi = true;
> > set beresp.ttl = 1m;
> >
> > # set beresp.grace = 24h;
> > }
> >
> >
> > Any helpful thoughts and or possible leads on this will be much
> appreciated.
> > kind regards,
> > -LC
> >
> > _______________________________________________
> > varnish-misc mailing list
> > varnish-misc at varnish-cache.org
> > https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
> -------------- next part --------------
> An HTML attachment was scrubbed...
> URL: <
> https://www.varnish-cache.org/lists/pipermail/varnish-misc/attachments/20140417/cf00e773/attachment-0001.html
> >
>
> ------------------------------
>
> _______________________________________________
> varnish-misc mailing list
> varnish-misc at varnish-cache.org
> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
>
> End of varnish-misc Digest, Vol 97, Issue 16
> ********************************************
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/varnish-misc/attachments/20140417/2fed476d/attachment.html>
More information about the varnish-misc
mailing list