Too many 503 backend errors
Anand Shah
imanandshah at gmail.com
Thu Dec 23 06:26:21 CET 2010
Thanks robert,
But i have already tried all this stuff's. The problem with probing is that
if your back-end is unhealthy for a second and if varnish declares it
unhealthy due to some reason than it shows backend unhealthy for next 5 mins
or so whereas the interval is only 5s. So if i m running on one back-end all
the requests for this backend fails.
The one i m using is a very domain with 3000 req/s.
so cannot afford a second of unhealthy network or unreachability.
Regards,
Anand
On Wed, Dec 22, 2010 at 4:28 PM, Robert Shilston <rtshilston at gmail.com>wrote:
> Anand,
>
> I've simplified our VCL as much as practical to demonstrate how we use the
> backends.
>
> > backend server2 {
> > .host = "a.b.c.12";
> > .port = "80";
> > .probe = {
> > .request =
> > "GET /healthforvarnish HTTP/1.1"
> > "Host: mysite.com"
> > "User-agent: Varnish-probe"
> > "Connection: close";
> > .interval = 5s;
> > .timeout = 2500ms;
> > .window = 10;
> > .threshold = 8;
> > }
> > }
> > backend server3 {
> > .host = "a.b.c.13";
> > .port = "80";
> > .probe = {
> > .request =
> > "GET /healthforvarnish HTTP/1.1"
> > "Host: mysite.com"
> > "User-agent: Varnish-probe"
> > "Connection: close";
> > .interval = 5s;
> > .timeout = 2500ms;
> > .window = 10;
> > .threshold = 8;
> > }
> > }
> > ...
> > backend server2failsafe { .host = "a.b.c.12"; .port="80"; }
> > backend server3failsafe { .host = "a.b.c.13"; .port="80"; }
> > ...
> >
> > director workingpool random {
> > { .backend = server2; .weight = 1; }
> > { .backend = server3; .weight = 1; }
> > ...
> > }
> >
> > director failsafepool random {
> > { .backend = server2failsafe; .weight = 1; }
> > { .backend = server3failsafe; .weight = 1; }
> > ...
> > }
> >
> > ...
> >
> > sub vcl_recv {
> > set req.backend = workingpool;
> >
> > if (!req.backend.healthy) {
> > set req.backend = failsafepool;
> > }
> >
>
>
>
> If you implement something like this, then run varnishncsa, you'll see
> which backends are being selected, and should deliver responses for 100% of
> your requests. If it does, then you'll need to debug the health checks to
> see why your backends might be being considered unhealthy.
>
> Rob
>
>
> On 22 Dec 2010, at 10:52, Anand Shah wrote:
>
> > It works for 6 out of 10 requests. I have also found that varnish
> > polls backend before sending a GET request. Since i never received any
> > request on my apache server for this dropped requests i am in doubt as
> > if what makes varnish feel the backend is in unhealthy state.
> >
> > Is there any way that i can troubleshoot?
> >
> >
> >
> > Anand
> >
> > On 12/22/10, Robert Shilston <rtshilston at gmail.com> wrote:
> >> Anand,
> >>
> >> Check that vcl_recv is actually setting a backend. It might be that
> there's
> >> a route through the VCL that doesn't cause a backend to be specified.
> >>
> >> Rob
> >>
> >> On 22 Dec 2010, at 08:39, Anand Shah wrote:
> >>
> >>> I immediately fired a test call using CURL which indicated me a
> >>> tcp_miss and i also got the response from origin.
> >>>
> >>> So backend was working and in healthy state.
> >>>
> >>>
> >>>
> >>> [root at mymachine ~]# curl --head --header host:imads.mydomain.com
> >>> "
> http://localhost//0/OasDefault/IIPM_CPL_Nov10_B3_MW/iipm_winter02_300x600_10dec_03.gif
> "
> >>> HTTP/1.1 200 OK
> >>> Last-Modified: Thu, 02 Dec 2010 12:19:14 GMT
> >>> P3P: CP="NON NID PSAa PSDa OUR IND UNI COM NAV
> >>> STA",policyref="/w3c/p3p.xml"
> >>> Content-Type: image/gif
> >>> cache-control: max-age=604800
> >>> Content-Length: 4150
> >>> Date: Wed, 22 Dec 2010 06:39:26 GMT
> >>> Connection: keep-alive
> >>> H-Served-By: mymachine
> >>> Server: mydomain/2.0.6
> >>> X-Cache: TCP_MISS
> >>>
> >>> [root at mymachine ~]# curl --head --header host:imads.mydomain.com
> >>> "
> http://localhost//0/OasDefault/IIPM_CPL_Nov10_B3_MW/iipm_winter02_300x600_10dec_03.gif
> "
> >>> HTTP/1.1 200 OK
> >>> Last-Modified: Thu, 02 Dec 2010 12:19:14 GMT
> >>> P3P: CP="NON NID PSAa PSDa OUR IND UNI COM NAV
> >>> STA",policyref="/w3c/p3p.xml"
> >>> Content-Type: image/gif
> >>> cache-control: max-age=604800
> >>> Content-Length: 4150
> >>> Date: Wed, 22 Dec 2010 06:39:33 GMT
> >>> Connection: keep-alive
> >>> H-Served-By: mymachine
> >>> Server: mydomain/2.0.6
> >>> X-Cache: TCP_HIT
> >>>
> >>>
> >>>
> >>>
> >>> Regards,
> >>> Anand
> >>>
> >>> On Wed, Dec 22, 2010 at 12:42:24PM +0530, Anand Shah wrote:
> >>>> I am running varnish on 64 bit Centos. Getting too many 503 errors
> >>>> recently and i tried debugging it with some help from Forums on
> >>>> mailinglist.
> >>>
> >>>> VARNISHLOGS:
> >>>>
> >>>> 180 VCL_call c miss fetch
> >>>> 180 FetchError c no backend connection
> >>>> 180 VCL_call c error deliver
> >>>
> >>> In my experience, "no backend connection" indicates that whatever
> >>> backend was selected to handle this request was unavailable due to the
> >>> health check/probe returning an error.
> >>>
> >>> What was the backend health at the time? You can use the 'debug.health'
> >>> command in the CLI to view health information, or
> >>>
> >>> varnishlog -i Backend_health
> >>>
> >>> to see the results of the health checks as they occur. If a backend is
> >>> marked as 'sick' then Varnish won't send any requests to it, since
> >>> it's unlikely to help the situation.
> >>>
> >>> _______________________________________________
> >>> varnish-misc mailing list
> >>> varnish-misc at varnish-cache.org
> >>> http://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
> >>>
> >>> _______________________________________________
> >>> varnish-misc mailing list
> >>> varnish-misc at varnish-cache.org
> >>> http://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
> >>
> >>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/varnish-misc/attachments/20101223/5ac536b1/attachment-0003.html>
More information about the varnish-misc
mailing list