[Varnish] #1151: Varnish to Apache sometimes return error 503

Varnish varnish-bugs at varnish-cache.org
Thu Jun 7 16:06:53 CEST 2012


#1151: Varnish to Apache sometimes return error 503
-----------------------------------+----------------------------------------
 Reporter:  rj@…                   |        Type:  defect
   Status:  new                    |    Priority:  high  
Milestone:                         |   Component:  build 
  Version:  3.0.2                  |    Severity:  major 
 Keywords:  503 http error apache  |  
-----------------------------------+----------------------------------------
 Hey guys.

 I have a setup where the firewall load balances down to two servers(load
 balancers). Here Ngingx parses http and https to a varnish cache on the
 same server. From the varnish it is sent the other varnish(the other load
 balance server) and from there to a backend server with apache2. (The
 setup was discussed with Martin Blix Grydeland during a varnish course
 held in Denmark and is working good besides the issue at hand)

 Now I have for some time been tracking some strange 503 errors. But I cant
 seem to find the silver bullet.

 Currently I am logging the 503 errors through varnish this way:

 {{{
 sudo varnishlog -c -m TxStatus:503 >> /home/rj/varnishlog503.log
 }}}

 and then referring to the apache access log to see if any 503 requests
 have been handled.

 Today I had a health check from the firewall that failed:

 {{{

    20 SessionOpen  c 127.0.0.1 34319 :8081
    20 ReqStart     c 127.0.0.1 34319 607335635
    20 RxRequest    c HEAD
    20 RxURL        c /health-check
    20 RxProtocol   c HTTP/1.0
    20 RxHeader     c X-Real-IP: 192.168.3.254
    20 RxHeader     c Host: 192.168.3.189
    20 RxHeader     c X-Forwarded-For: 192.168.3.254
    20 RxHeader     c Connection: close
    20 RxHeader     c User-Agent: Astaro Service Monitor 0.9
    20 RxHeader     c Accept: */*
    20 VCL_call     c recv lookup
    20 VCL_call     c hash
    20 Hash         c /health-check
    20 VCL_return   c hash
    20 VCL_call     c miss fetch
    20 Backend      c 33 aurum aurum
    20 FetchError   c http first read error: -1 11 (No error recorded)
    20 VCL_call     c error deliver
    20 VCL_call     c deliver deliver
    20 TxProtocol   c HTTP/1.1
    20 TxStatus     c 503
    20 TxResponse   c Service Unavailable
    20 TxHeader     c Server: Varnish
    20 TxHeader     c Content-Type: text/html; charset=utf-8
    20 TxHeader     c Retry-After: 5
    20 TxHeader     c Content-Length: 879
    20 TxHeader     c Accept-Ranges: bytes
    20 TxHeader     c Date: Wed, 06 Jun 2012 12:35:12 GMT
    20 TxHeader     c X-Varnish: 607335635
    20 TxHeader     c Age: 60
    20 TxHeader     c Via: 1.1 varnish
    20 TxHeader     c Connection: close
    20 Length       c 879
    20 ReqEnd       c 607335635 1338986052.649786949 1338986112.648169994
 0.000160217 59.997980356 0.000402689
 }}}

 And here is another example of a 503 error:

 {{{

    16 SessionOpen  c 127.0.0.1 44997 :8081
    16 ReqStart     c 127.0.0.1 44997 613052041
    16 RxRequest    c POST
    16 RxURL        c /?id=412
    16 RxProtocol   c HTTP/1.0
    16 RxHeader     c X-Real-IP: 80.210.231.31
    16 RxHeader     c Host: gnisten.sfoweb.dk
    16 RxHeader     c X-Forwarded-For: 80.210.231.31
    16 RxHeader     c Connection: close
    16 RxHeader     c Content-Length: 434
    16 RxHeader     c Cache-Control: max-age=0
    16 RxHeader     c Origin: https://gnisten.sfoweb.dk
    16 RxHeader     c User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64)
 AppleWebKit/536.5 (KHTML, like Gecko) Chrome/19.0.1084.52 Safari/536.5
    16 RxHeader     c Content-Type: application/x-www-form-urlencoded
    16 RxHeader     c Accept:
 text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
    16 RxHeader     c Referer: https://gnisten.sfoweb.dk/?id=412
    16 RxHeader     c Accept-Encoding: gzip,deflate,sdch
    16 RxHeader     c Accept-Language: da-DK,da;q=0.8,en-US;q=0.6,en;q=0.4
    16 RxHeader     c Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.3
    16 RxHeader     c Cookie: PHPSESSID=ovqafgba8q4pvokmu3eso89ct4;
 fe_typo_user=9ea6d60afb41a72033479726ae4fac89
    16 VCL_call     c recv pass
    16 VCL_call     c hash
    16 Hash         c /?id=412
    16 VCL_return   c hash
    16 VCL_call     c pass pass
    16 FetchError   c no backend connection
    16 VCL_call     c error deliver
    16 TxProtocol   c HTTP/1.1
    16 TxStatus     c 503
    16 TxResponse   c Service Unavailable
    16 TxHeader     c Server: Varnish
    16 TxHeader     c Content-Type: text/html; charset=utf-8
    16 TxHeader     c Retry-After: 5
    16 TxHeader     c Content-Length: 879
    16 TxHeader     c Accept-Ranges: bytes
    16 TxHeader     c Date: Thu, 07 Jun 2012 13:34:41 GMT
    16 TxHeader     c X-Varnish: 613052041
    16 TxHeader     c Age: 1
    16 TxHeader     c Via: 1.1 varnish
    16 TxHeader     c Connection: close
    16 Length       c 879
    16 ReqEnd       c 613052041 1339076080.814462662 1339076081.515620232
 0.000342369 0.701060295 0.000097275

 }}}


 Now the backend server (apache) does not have any 503 in the access log at
 this point. So it would seem the request never reached the server. At the
 time where I get the 503 error traffic is still being sent to the server
 (I can see requests in the access log - but not the 503 one). Even in the
 morning when the server dosen't seem to be doing anything.

 A summery of the FetchErrors for the last two days:

 {{{
     7 FetchError   c http first read error: -1 11 (No error recorded)
     4 FetchError   c http first read error: -1 11 (No error recorded)
     4 FetchError   c http first read error: -1 11 (No error recorded)
     6 FetchError   c http first read error: -1 11 (No error recorded)
    24 FetchError   c http first read error: -1 11 (No error recorded)
     4 FetchError   c http first read error: -1 11 (No error recorded)
     6 FetchError   c http first read error: -1 11 (No error recorded)
    20 FetchError   c http first read error: -1 11 (No error recorded)
    20 FetchError   c http first read error: -1 11 (No error recorded)
    19 FetchError   c http first read error: -1 11 (No error recorded)
     4 FetchError   c http first read error: -1 11 (No error recorded)
     4 FetchError   c http first read error: -1 11 (No error recorded)
     4 FetchError   c http first read error: -1 11 (No error recorded)
    23 FetchError   c http first read error: -1 11 (No error recorded)
     4 FetchError   c http first read error: -1 11 (No error recorded)
    18 FetchError   c http first read error: -1 11 (No error recorded)
    36 FetchError   c no backend connection
    18 FetchError   c http first read error: -1 11 (No error recorded)
    24 FetchError   c http first read error: -1 11 (No error recorded)
    22 FetchError   c http first read error: -1 11 (No error recorded)
    23 FetchError   c no backend connection
    33 FetchError   c no backend connection
    25 FetchError   c http first read error: -1 11 (No error recorded)
    30 FetchError   c http first read error: -1 11 (No error recorded)
    36 FetchError   c no backend connection
    17 FetchError   c http first read error: -1 11 (No error recorded)
     5 FetchError   c http first read error: -1 11 (No error recorded)
    17 FetchError   c http first read error: -1 11 (No error recorded)
    34 FetchError   c http first read error: -1 11 (No error recorded)
    26 FetchError   c http first read error: -1 11 (No error recorded)
    20 FetchError   c http first read error: -1 11 (No error recorded)
     5 FetchError   c http first read error: -1 11 (No error recorded)
    17 FetchError   c http first read error: -1 11 (No error recorded)
    19 FetchError   c http first read error: -1 11 (No error recorded)
     7 FetchError   c http first read error: -1 11 (No error recorded)
     7 FetchError   c http first read error: -1 11 (No error recorded)
    18 FetchError   c http first read error: -1 11 (No error recorded)
     7 FetchError   c http first read error: -1 11 (No error recorded)
     4 FetchError   c http first read error: -1 11 (No error recorded)
     4 FetchError   c http first read error: -1 11 (No error recorded)
     7 FetchError   c http first read error: -1 11 (No error recorded)
     4 FetchError   c http first read error: -1 11 (No error recorded)
    17 FetchError   c http first read error: -1 11 (No error recorded)
    20 FetchError   c http first read error: -1 11 (No error recorded)
     6 FetchError   c http first read error: -1 11 (No error recorded)
    19 FetchError   c http first read error: -1 11 (No error recorded)
     4 FetchError   c http first read error: -1 11 (No error recorded)
     4 FetchError   c http first read error: -1 11 (No error recorded)
     4 FetchError   c http first read error: -1 11 (No error recorded)
     4 FetchError   c http first read error: -1 11 (No error recorded)
     4 FetchError   c http first read error: -1 11 (No error recorded)
     7 FetchError   c http first read error: -1 11 (No error recorded)
     7 FetchError   c backend write error: 11 (Resource temporarily
 unavailable)
     4 FetchError   c http first read error: -1 11 (No error recorded)
    32 FetchError   c http first read error: -1 11 (No error recorded)
    18 FetchError   c no backend connection
     7 FetchError   c no backend connection
     4 FetchError   c no backend connection
 }}}

 I haven't changed the default timeout values for varnish. This is my
 configuration for the backends and the director.


 {{{

 backend radon {
         .host = "192.168.3.186";
         .port = "80";
         .probe = {
                 .url = "/health-check/";
                 .interval = 3s;
                 .window = 5;
                 .threshold = 2;
         }
 }

 backend xenon {
         .host = "192.168.3.187";
         .port = "80";
         .probe = {
                 .url = "/health-check/";
                 .interval = 3s;
                 .window = 5;
                 .threshold = 2;
         }
 }

 backend aurum {
         .host = "127.0.0.1";
         .port = "8085";
         .probe = {
                 .url = "/health-check";
                 .interval = 3s;
                 .window = 5;
                 .threshold = 2;
         }
 }

 backend iridium {
         .host = "192.168.3.189";
         .port = "8081";
         .probe = {
                 .url = "/varnish-health";
                 .interval = 3s;
                 .window = 5;
                 .threshold = 2;
         }
 }

 director balance client {
         {
                 .backend = radon;
                 .weight = 1;
         }
         {
                 .backend = xenon;
                 .weight = 1;
         }
 }

 }}}

 I have also tried to double the number of threads apache is allowed to run
 but without any result. I still get around 10-20 503 error a day and they
 come from various requests.

 I'm assuming this is some kind of defect/bug. I haven't been able to find
 anything about the issues in the Internet. I have set the priority and
 severity as a bit higher than normal since out clients receive 503 error
 in delicate operations.

 Best regards Ronnie

-- 
Ticket URL: <https://www.varnish-cache.org/trac/ticket/1151>
Varnish <https://varnish-cache.org/>
The Varnish HTTP Accelerator




More information about the varnish-bugs mailing list