503 error from varnish

[Tim Dunphy]

Hey guys,

 I've setup my wiki to be delivered through varnish. My first breakthrough
in getting this to work was to allow for http basic auth to pass through
varnish so the user will be prompted to login. This is working quite well
at this point.

But my current problem is that I've been able to get the main page to the
site cached, and a few others. But when I click to like the 4th or 5th page
on the wiki that I want to cache through varnish, everything comes to a
halt. I get a 503 - service unavailable error, instead of varnish going to
the web server to retrieve the page I had clicked on.

In my varnishlog, I'm seeing the following which results in an error:

10 SessionOpen  c 10.10.10.25 35852 :80

   10 ReqStart     c 10.10.10.25 35852 1369002554

   10 RxRequest    c GET

   10 RxURL        c /index.php/Affluencers

   10 RxProtocol   c HTTP/1.1

   10 RxHeader     c Host: wiki.example.com

   10 RxHeader     c Cache-Control: max-age=0

   10 RxHeader     c Authorization: Basic YWRtaW46RHVrMzBmWmgwdQ==

   10 RxHeader     c Accept:
text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8

   10 RxHeader     c User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X
10_10_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/43.0.2357.130
Safari/537.36

   10 RxHeader     c Referer: http://wiki.example.com/index.php/NBCUniversal

   10 RxHeader     c Accept-Encoding: gzip, deflate, sdch

   10 RxHeader     c Accept-Language: en-US,en;q=0.8

   10 RxHeader     c X-Forwarded-For: 47.18.111.100

   10 RxHeader     c Connection: close

   10 VCL_call     c recv lookup

   10 VCL_call     c hash

   10 Hash         c /index.php/Affluencers

   10 Hash         c wiki.example.com

   10 VCL_return   c hash

   10 VCL_call     c miss fetch

   10 FetchError   c no backend connection

   10 VCL_call     c error deliver

   10 VCL_call     c deliver deliver

   10 TxProtocol   c HTTP/1.1

   10 TxStatus     c 503

   10 TxResponse   c Service Unavailable

   10 TxHeader     c Server: Varnish

   10 TxHeader     c Content-Type: text/html; charset=utf-8

   10 TxHeader     c Retry-After: 5

   10 TxHeader     c Content-Length: 419

   10 TxHeader     c Accept-Ranges: bytes

   10 TxHeader     c Date: Sun, 05 Jul 2015 00:42:16 GMT

   10 TxHeader     c X-Varnish: 1369002554

   10 TxHeader     c Age: 0

   10 TxHeader     c Via: 1.1 varnish

   10 TxHeader     c Connection: close

   10 TxHeader     c X-Cache: MISS

   10 Length       c 419

   10 ReqEnd       c 1369002554 1436056936.048977137 1436056936.049291372
0.000086308 0.000152111 0.000162125

   10 SessionClose c error

   10 StatSess     c 10.10.10.25 35852 0 1 1 0 0 0 272 419

I'm not sure how to interpret that to be honest. But I'd like some help
resolving this issue. :)

One thing that I considered is that I may be running out of memory while
trying to cache the site.

The varnish host itself is only a 1GB VM with 40GB of disk space.

Here's my varnish system config:

grep -v '#' /etc/sysconfig/varnish

NFILES=131072

MEMLOCK=82000

NPROCS="unlimited"

RELOAD_VCL=1

VARNISH_VCL_CONF=/etc/varnish/default.vcl

VARNISH_LISTEN_PORT=80

VARNISH_ADMIN_LISTEN_ADDRESS=127.0.0.1

VARNISH_ADMIN_LISTEN_PORT=6082

VARNISH_SECRET_FILE=/etc/varnish/secret

VARNISH_MIN_THREADS=50

VARNISH_MAX_THREADS=1000

VARNISH_THREAD_TIMEOUT=120

VARNISH_STORAGE_FILE=/var/lib/varnish/varnish_storage.bin

VARNISH_STORAGE_SIZE=10G

VARNISH_STORAGE="malloc,${VARNISH_STORAGE_SIZE}"

VARNISH_TTL=120

DAEMON_OPTS="-a ${VARNISH_LISTEN_ADDRESS}:${VARNISH_LISTEN_PORT} \

             -f ${VARNISH_VCL_CONF} \

             -T
${VARNISH_ADMIN_LISTEN_ADDRESS}:${VARNISH_ADMIN_LISTEN_PORT} \

             -t ${VARNISH_TTL} \

             -w
${VARNISH_MIN_THREADS},${VARNISH_MAX_THREADS},${VARNISH_THREAD_TIMEOUT} \

             -u varnish -g varnish \

             -S ${VARNISH_SECRET_FILE} \


             -s ${VARNISH_STORAGE}"

And this is my very simple VCL file:

backend web1 {

  .host = "10.10.10.25";

  .port = "80";

  .connect_timeout = 30s;

  .first_byte_timeout = 30s;

  .between_bytes_timeout = 30s;

  .max_connections = 70;

  .probe = {

  .request =

   "GET /healthcheck.php HTTP/1.1"

   "Host: wiki.example.com"

   "Connection: close";

   .interval = 10m;

   .timeout = 60s;

   .window = 3;

   .threshold = 2;

   }

}

backend web2 {

  .host = "10.10.10.26";

  .port = "80";

  .connect_timeout = 30s;

  .first_byte_timeout = 30s;

  .between_bytes_timeout = 30s;

  .max_connections = 70;

  .probe = {

  .request =

   "GET /healthcheck.php HTTP/1.1"

   "Host: wiki.example.com"

   "Connection: close";

   .interval = 10m;

   .timeout = 60s;

   .window = 3;

   .threshold = 2;

   }

}


director www round-robin {

  { .backend = web1;   }

  { .backend = web2;  }

 }


sub vcl_recv {

    set req.backend = www;
    return (lookup);

}

sub vcl_fetch {

      set beresp.ttl = 3600s;

      set beresp.grace = 4h;

      return (deliver);

}

sub vcl_deliver {

     if (obj.hits> 0) {

      set resp.http.X-Cache = "HIT";

     } else {

        set resp.http.X-Cache = "MISS";

     }

 }

So I'm wondering if I'm anywhere near the mark that I could be running out
of memory when trying to cache pages to my wiki site.

I'd really appreciate any feedback you guys may have!

Thanks,

Tim
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/varnish-misc/attachments/20150704/548b070e/attachment.html>