Too many 503 backend errors

[Robert Shilston]

Anand, 

I've simplified our VCL as much as practical to demonstrate how we use the backends.  

> backend server2 {
>     .host = "a.b.c.12";
>     .port = "80";
>     .probe = {
>         .request =
>             "GET /healthforvarnish HTTP/1.1"
>             "Host: mysite.com"
>             "User-agent: Varnish-probe"
>             "Connection: close";
>         .interval = 5s;
>         .timeout = 2500ms;
>         .window = 10;
>         .threshold = 8;
>     }
> }
> backend server3 {
>     .host = "a.b.c.13";
>     .port = "80";
>     .probe = {
>         .request =
>             "GET /healthforvarnish HTTP/1.1"
>             "Host: mysite.com"
>             "User-agent: Varnish-probe"
>             "Connection: close";
>         .interval = 5s;
>         .timeout = 2500ms;
>         .window = 10;
>         .threshold = 8;
>     }
> }
> ...
> backend server2failsafe { .host = "a.b.c.12"; .port="80"; }
> backend server3failsafe { .host = "a.b.c.13"; .port="80"; }
> ...
> 
> director workingpool random {
>     { .backend = server2; .weight = 1; }
>     { .backend = server3; .weight = 1; }
>     ...
> }
> 
> director failsafepool random {
>     { .backend = server2failsafe; .weight = 1; }
>     { .backend = server3failsafe; .weight = 1; }
>     ...
> }
> 
> ...
> 
> sub vcl_recv {
>     set req.backend = workingpool;
> 
>     if (!req.backend.healthy) {
>         set req.backend = failsafepool;
>     }
> 



If you implement something like this, then run varnishncsa, you'll see which backends are being selected, and should deliver responses for 100% of your requests.  If it does, then you'll need to debug the health checks to see why your backends might be being considered unhealthy.

Rob


On 22 Dec 2010, at 10:52, Anand Shah wrote:

> It works for 6 out of 10 requests. I have also found that varnish
> polls backend before sending a GET request. Since i never received any
> request on my apache server for this dropped requests i am in doubt as
> if what makes varnish feel the backend is in unhealthy state.
> 
> Is there any way that i can troubleshoot?
> 
> 
> 
> Anand
> 
> On 12/22/10, Robert Shilston <rtshilston at gmail.com> wrote:
>> Anand,
>> 
>> Check that vcl_recv is actually setting a backend.  It might be that there's
>> a route through the VCL that doesn't cause a backend to be specified.
>> 
>> Rob
>> 
>> On 22 Dec 2010, at 08:39, Anand Shah wrote:
>> 
>>> I immediately fired a test call using CURL which indicated me a
>>> tcp_miss and i also got the response from origin.
>>> 
>>> So backend was working and in healthy state.
>>> 
>>> 
>>> 
>>> [root at mymachine ~]# curl --head --header host:imads.mydomain.com
>>> "http://localhost//0/OasDefault/IIPM_CPL_Nov10_B3_MW/iipm_winter02_300x600_10dec_03.gif"
>>> HTTP/1.1 200 OK
>>> Last-Modified: Thu, 02 Dec 2010 12:19:14 GMT
>>> P3P: CP="NON NID PSAa PSDa OUR IND UNI COM NAV
>>> STA",policyref="/w3c/p3p.xml"
>>> Content-Type: image/gif
>>> cache-control: max-age=604800
>>> Content-Length: 4150
>>> Date: Wed, 22 Dec 2010 06:39:26 GMT
>>> Connection: keep-alive
>>> H-Served-By: mymachine
>>> Server: mydomain/2.0.6
>>> X-Cache: TCP_MISS
>>> 
>>> [root at mymachine ~]# curl --head --header host:imads.mydomain.com
>>> "http://localhost//0/OasDefault/IIPM_CPL_Nov10_B3_MW/iipm_winter02_300x600_10dec_03.gif"
>>> HTTP/1.1 200 OK
>>> Last-Modified: Thu, 02 Dec 2010 12:19:14 GMT
>>> P3P: CP="NON NID PSAa PSDa OUR IND UNI COM NAV
>>> STA",policyref="/w3c/p3p.xml"
>>> Content-Type: image/gif
>>> cache-control: max-age=604800
>>> Content-Length: 4150
>>> Date: Wed, 22 Dec 2010 06:39:33 GMT
>>> Connection: keep-alive
>>> H-Served-By: mymachine
>>> Server: mydomain/2.0.6
>>> X-Cache: TCP_HIT
>>> 
>>> 
>>> 
>>> 
>>> Regards,
>>>  Anand
>>> 
>>> On Wed, Dec 22, 2010 at 12:42:24PM +0530, Anand Shah wrote:
>>>> I am running varnish on 64 bit Centos. Getting too many 503 errors
>>>> recently and i tried debugging it with some help from Forums on
>>>> mailinglist.
>>> 
>>>> VARNISHLOGS:
>>>> 
>>>> 180 VCL_call     c miss fetch
>>>> 180 FetchError   c no backend connection
>>>> 180 VCL_call     c error deliver
>>> 
>>> In my experience, "no backend connection" indicates that whatever
>>> backend was selected to handle this request was unavailable due to the
>>> health check/probe returning an error.
>>> 
>>> What was the backend health at the time? You can use the 'debug.health'
>>> command in the CLI to view health information, or
>>> 
>>> varnishlog -i Backend_health
>>> 
>>> to see the results of the health checks as they occur. If a backend is
>>> marked as 'sick' then Varnish won't send any requests to it, since
>>> it's unlikely to help the situation.
>>> 
>>> _______________________________________________
>>> varnish-misc mailing list
>>> varnish-misc at varnish-cache.org
>>> http://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
>>> 
>>> _______________________________________________
>>> varnish-misc mailing list
>>> varnish-misc at varnish-cache.org
>>> http://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
>> 
>>