frequent 504's on VCL

[Tim Dunphy]

Hey guys,

I'm getting frequent 504 errors in the browser when using even this
stripped down version of my VCL. I'm also seeing the pages load with the
css totally blown and just the text showing up with no graphics and no
formatting.

It's pretty odd how inconsistent this VCL is behaving.

On one load of the page where the graphics are broken I'm seeing this error
in the logs:

   10 TxHeader     c Connection: close
   10 TxHeader     c X-Cache: MISS
   10 Debug        c Write error, retval = -1, len = 602, errno =
Connection reset by peer
   10 ReqEnd       c 241437586 1438049428.763690233 1438049435.868994236
0.000110626 7.105183125 0.000120878
   10 StatSess     c 54.86.143.49 48979 7 1 1 0 0 1 602 0
    0 CLI          - Rd ping
    0 CLI          - Wr 200 19 PONG 1438049436 1.0

And on 504 errors I'm seeing this result in the logs:

10 TxHeader     c Via: 1.1 varnish
   10 TxHeader     c Connection: close
   10 TxHeader     c X-Cache: MISS
   10 Length       c 316
   10 ReqEnd       c 241437672 1438049679.750560999 1438049679.750730276
0.000099182 0.000094652 0.000074625
   10 SessionClose c error
   10 StatSess     c 54.86.143.49 49821 0 1 1 0 0 0 285 316


I'm running 3 back ends using apache 2.4 on Centos 7. I'm running two
Varnish nodes at version 3.0.5.

All 3 web backends are checking in as healthy:

[root at varnish1:/etc/varnish] #varnishadm -n  varnish1   debug.health
Backend web1 is Healthy
Current states  good:  3 threshold:  2 window:  3
Average responsetime of good probes: 0.026873
Oldest                                                    Newest
================================================================
--------------------------------------------------------------44 Good IPv4
--------------------------------------------------------------XX Good Xmit
--------------------------------------------------------------RR Good Recv
------------------------------------------------------------HHHH Happy
Backend web2 is Healthy
Current states  good:  3 threshold:  2 window:  3
Average responsetime of good probes: 0.029118
Oldest                                                    Newest
================================================================
--------------------------------------------------------------44 Good IPv4
--------------------------------------------------------------XX Good Xmit
--------------------------------------------------------------RR Good Recv
------------------------------------------------------------HHHH Happy
Backend web3 is Healthy
Current states  good:  3 threshold:  2 window:  3
Average responsetime of good probes: 0.029101
Oldest                                                    Newest
================================================================
--------------------------------------------------------------44 Good IPv4
--------------------------------------------------------------XX Good Xmit
--------------------------------------------------------------RR Good Recv
------------------------------------------------------------HHHH Happy

And here is the VCL that I am having this trouble with:

[root at varnish1:/etc/varnish] #egrep -v "^#|^$" default.vcl

backend web1 {

  .host = “10.10.10.25”;

  .port = "80";

  .connect_timeout = 45s;

  .first_byte_timeout = 45s;

  .between_bytes_timeout = 45s;

  .max_connections = 70;

  .probe = {

  .request =

   "GET /healthcheck.php HTTP/1.1"

   "Host: wiki.example.com"

   "Authorization: Basic SomeLongBase64Hash="

   "Connection: close";

   .interval = 10m;

   .timeout = 60s;

   .window = 3;

   .threshold = 2;

   }

}

backend web2 {

  .host = “10.10.10.26”;

  .port = "80";

  .connect_timeout = 45s;

  .first_byte_timeout = 45s;

  .between_bytes_timeout = 45s;

  .max_connections = 70;

  .probe = {

  .request =

   "GET /healthcheck.php HTTP/1.1"

   "Host: wiki.example.com"

   "Authorization: Basic SomeLongBase64Hash="

   "Connection: close";

   .interval = 10m;

   .timeout = 60s;

   .window = 3;

   .threshold = 2;

   }

}

backend web3 {

  .host = “10.10.10.27”;

  .port = "80";

  .connect_timeout = 45s;

  .first_byte_timeout = 45s;

  .between_bytes_timeout = 45s;

  .max_connections = 70;

  .probe = {

  .request =

   "GET /healthcheck.php HTTP/1.1"

   "Host: wiki.example.com"

   "Authorization: Basic SomeLongBase64Hash="

   "Connection: close";

   .interval = 10m;

   .timeout = 60s;

   .window = 3;

   .threshold = 2;

   }

}

director www round-robin {

  { .backend = web1;  }

  { .backend = web2;  }

  { .backend = web3;  }

 }

sub vcl_recv {

    set req.backend = www;

    unset req.http.Cookie;

     if (! req.http.Authorization ~ "Basic SomeLongBase64Hash=" && !
req.http.Authorization ~ "Basic AnotherLongBase64Hash=" )

      {

       error 401 "Restricted";

     }

    if (req.url ~ "&action=submit($|/)") {

        return (pass);

    }

    return (lookup);

}

sub vcl_fetch {

      set beresp.ttl = 3600s;

      set beresp.grace = 4h;

      return (deliver);

}

sub vcl_error {

  if (obj.status == 401) {

  set obj.http.Content-Type = "text/html; charset=utf-8";

  set obj.http.WWW-Authenticate = "Basic realm=Secured";

  synthetic {"

   <!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN"  "
http://www.w3.org/TR/1999/REC-html401-19991224/loose.dtd">

    <HTML>

    <HEAD>

    <TITLE>Error</TITLE>

    <META HTTP-EQUIV='Content-Type' CONTENT='text/html;'>

    </HEAD>

    <BODY><H1>401 Unauthorized (varnish)</H1></BODY>

    </HTML>

    "};

     return (deliver);

    }

}

sub vcl_deliver {

     if (obj.hits> 0) {

      set resp.http.X-Cache = "HIT";

     } else {

        set resp.http.X-Cache = "MISS";

     }

 }


The IP's you see listed above are fake. I'm not really running the web
servers on a 10-net.

I'm really looking forward to getting this solved! And I'd appreciate any
feedback you may have.


Thanks,

Tim



-- 
GPG me!!

gpg --keyserver pool.sks-keyservers.net --recv-keys F186197B
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/varnish-misc/attachments/20150727/5e231229/attachment.html>