503 service unavailable error

Wed Jul 8 18:14:03 CEST 2015

Hi guys,

 I'm having an issue where my varnish server will stop working after a
while of browsing around the site I'm using it with and throw a 503 server
unavailable error.

In my varnish logs I'm getting a 'no backend connection error':

  * 10 FetchError   c no backend connection*
   10 VCL_call     c error deliver
   10 VCL_call     c deliver deliver
   10 TxProtocol   c HTTP/1.1
   10 TxStatus     c 503
   10 TxResponse   c Service Unavailable
   10 TxHeader     c Server: Varnish

And if I do a GET on the healthcheck from the command line on the varnish
server, I get a 503 response from varnish:

#GET http://wiki.example.com/healthcheck.php

<?xml version="1.0" encoding="utf-8"?>
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN"
 "http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">
<html>
  <head>
    <title>503 Service Unavailable</title>
  </head>
  <body>
    <h1>Error 503 Service Unavailable</h1>
    <p>Service Unavailable</p>
    <h3>Guru Meditation:</h3>
    <p>XID: 2107225059</p>
    <hr>
    <p>Varnish cache server</p>
  </body>
</html>

But if I do another GET on the healthcheck file from the varnish server to
another apache VHOST on the same server as the wiki site that responds to
the IP of the web server instead of the IP for the varnish server, the GET
works:

#GET http://ops1.example.com/healthcheck.php
good

So I'm not sure why varnish is having trouble reaching the HC file. The web
server is a little far from the varnish server. The varnish machines are in
NYC and the web servers are in northern Virginia.

So I tried setting the timeouts in the varnish config to a really high
number. And that was working for a while. But today I noticed that it
stopped working. I'll have to restart the varnish service and browse the
site for a while. Then it'll stop working again and produce the 503 error.
It's pretty annoying!

I was wondering if there might be something in my VCL I could tweak to make
this work? Or if the fact is that the web servers are simply too far from
varnish for this to be practical.

Here's my VCL file. It's pretty basic:

backend web1 {
  .host = "10.10.10.25";
  .port = "80";
  .connect_timeout = 1200s;
  .first_byte_timeout = 1200s;
  .between_bytes_timeout = 1200s;
  .max_connections = 70;
  .probe = {
  .request =
   "GET /healthcheck.php HTTP/1.1"
   "Host: wiki.example.com"
   "Connection: close";
   .interval = 10m;
   .timeout = 60s;
   .window = 3;
   .threshold = 2;
   }
}

backend web2 {
  .host = "10.10.10.26";
  .port = "80";
  .connect_timeout = 1200s;
  .first_byte_timeout = 1200s;
  .between_bytes_timeout = 1200s;
  .max_connections = 70;
  .probe = {
  .request =
   "GET /healthcheck.php HTTP/1.1"
   "Host: wiki.example.com"
   "Connection: close";
   .interval = 10m;
   .timeout = 60s;
   .window = 3;
   .threshold = 2;
   }
}

director www round-robin {
  { .backend = web1;   }
  { .backend = web2;  }
 }

sub vcl_recv {

    if (req.url ~ "&action=submit($|/)") {
        return (pass);
    }

    set req.backend = www;
    return (lookup);
}

sub vcl_fetch {
      set beresp.ttl = 3600s;
      set beresp.grace = 4h;
      return (deliver);
}

sub vcl_deliver {
     if (obj.hits> 0) {
      set resp.http.X-Cache = "HIT";
     } else {
        set resp.http.X-Cache = "MISS";
     }
 }

Thanks,
Tim

-- 
GPG me!!

gpg --keyserver pool.sks-keyservers.net --recv-keys F186197B
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/varnish-misc/attachments/20150708/a5e7800e/attachment.html>