[Varnish] #1151: Varnish to Apache sometimes return error 503
Varnish
varnish-bugs at varnish-cache.org
Thu Jun 7 16:06:53 CEST 2012
#1151: Varnish to Apache sometimes return error 503
-----------------------------------+----------------------------------------
Reporter: rj@… | Type: defect
Status: new | Priority: high
Milestone: | Component: build
Version: 3.0.2 | Severity: major
Keywords: 503 http error apache |
-----------------------------------+----------------------------------------
Hey guys.
I have a setup where the firewall load balances down to two servers(load
balancers). Here Ngingx parses http and https to a varnish cache on the
same server. From the varnish it is sent the other varnish(the other load
balance server) and from there to a backend server with apache2. (The
setup was discussed with Martin Blix Grydeland during a varnish course
held in Denmark and is working good besides the issue at hand)
Now I have for some time been tracking some strange 503 errors. But I cant
seem to find the silver bullet.
Currently I am logging the 503 errors through varnish this way:
{{{
sudo varnishlog -c -m TxStatus:503 >> /home/rj/varnishlog503.log
}}}
and then referring to the apache access log to see if any 503 requests
have been handled.
Today I had a health check from the firewall that failed:
{{{
20 SessionOpen c 127.0.0.1 34319 :8081
20 ReqStart c 127.0.0.1 34319 607335635
20 RxRequest c HEAD
20 RxURL c /health-check
20 RxProtocol c HTTP/1.0
20 RxHeader c X-Real-IP: 192.168.3.254
20 RxHeader c Host: 192.168.3.189
20 RxHeader c X-Forwarded-For: 192.168.3.254
20 RxHeader c Connection: close
20 RxHeader c User-Agent: Astaro Service Monitor 0.9
20 RxHeader c Accept: */*
20 VCL_call c recv lookup
20 VCL_call c hash
20 Hash c /health-check
20 VCL_return c hash
20 VCL_call c miss fetch
20 Backend c 33 aurum aurum
20 FetchError c http first read error: -1 11 (No error recorded)
20 VCL_call c error deliver
20 VCL_call c deliver deliver
20 TxProtocol c HTTP/1.1
20 TxStatus c 503
20 TxResponse c Service Unavailable
20 TxHeader c Server: Varnish
20 TxHeader c Content-Type: text/html; charset=utf-8
20 TxHeader c Retry-After: 5
20 TxHeader c Content-Length: 879
20 TxHeader c Accept-Ranges: bytes
20 TxHeader c Date: Wed, 06 Jun 2012 12:35:12 GMT
20 TxHeader c X-Varnish: 607335635
20 TxHeader c Age: 60
20 TxHeader c Via: 1.1 varnish
20 TxHeader c Connection: close
20 Length c 879
20 ReqEnd c 607335635 1338986052.649786949 1338986112.648169994
0.000160217 59.997980356 0.000402689
}}}
And here is another example of a 503 error:
{{{
16 SessionOpen c 127.0.0.1 44997 :8081
16 ReqStart c 127.0.0.1 44997 613052041
16 RxRequest c POST
16 RxURL c /?id=412
16 RxProtocol c HTTP/1.0
16 RxHeader c X-Real-IP: 80.210.231.31
16 RxHeader c Host: gnisten.sfoweb.dk
16 RxHeader c X-Forwarded-For: 80.210.231.31
16 RxHeader c Connection: close
16 RxHeader c Content-Length: 434
16 RxHeader c Cache-Control: max-age=0
16 RxHeader c Origin: https://gnisten.sfoweb.dk
16 RxHeader c User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64)
AppleWebKit/536.5 (KHTML, like Gecko) Chrome/19.0.1084.52 Safari/536.5
16 RxHeader c Content-Type: application/x-www-form-urlencoded
16 RxHeader c Accept:
text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
16 RxHeader c Referer: https://gnisten.sfoweb.dk/?id=412
16 RxHeader c Accept-Encoding: gzip,deflate,sdch
16 RxHeader c Accept-Language: da-DK,da;q=0.8,en-US;q=0.6,en;q=0.4
16 RxHeader c Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.3
16 RxHeader c Cookie: PHPSESSID=ovqafgba8q4pvokmu3eso89ct4;
fe_typo_user=9ea6d60afb41a72033479726ae4fac89
16 VCL_call c recv pass
16 VCL_call c hash
16 Hash c /?id=412
16 VCL_return c hash
16 VCL_call c pass pass
16 FetchError c no backend connection
16 VCL_call c error deliver
16 TxProtocol c HTTP/1.1
16 TxStatus c 503
16 TxResponse c Service Unavailable
16 TxHeader c Server: Varnish
16 TxHeader c Content-Type: text/html; charset=utf-8
16 TxHeader c Retry-After: 5
16 TxHeader c Content-Length: 879
16 TxHeader c Accept-Ranges: bytes
16 TxHeader c Date: Thu, 07 Jun 2012 13:34:41 GMT
16 TxHeader c X-Varnish: 613052041
16 TxHeader c Age: 1
16 TxHeader c Via: 1.1 varnish
16 TxHeader c Connection: close
16 Length c 879
16 ReqEnd c 613052041 1339076080.814462662 1339076081.515620232
0.000342369 0.701060295 0.000097275
}}}
Now the backend server (apache) does not have any 503 in the access log at
this point. So it would seem the request never reached the server. At the
time where I get the 503 error traffic is still being sent to the server
(I can see requests in the access log - but not the 503 one). Even in the
morning when the server dosen't seem to be doing anything.
A summery of the FetchErrors for the last two days:
{{{
7 FetchError c http first read error: -1 11 (No error recorded)
4 FetchError c http first read error: -1 11 (No error recorded)
4 FetchError c http first read error: -1 11 (No error recorded)
6 FetchError c http first read error: -1 11 (No error recorded)
24 FetchError c http first read error: -1 11 (No error recorded)
4 FetchError c http first read error: -1 11 (No error recorded)
6 FetchError c http first read error: -1 11 (No error recorded)
20 FetchError c http first read error: -1 11 (No error recorded)
20 FetchError c http first read error: -1 11 (No error recorded)
19 FetchError c http first read error: -1 11 (No error recorded)
4 FetchError c http first read error: -1 11 (No error recorded)
4 FetchError c http first read error: -1 11 (No error recorded)
4 FetchError c http first read error: -1 11 (No error recorded)
23 FetchError c http first read error: -1 11 (No error recorded)
4 FetchError c http first read error: -1 11 (No error recorded)
18 FetchError c http first read error: -1 11 (No error recorded)
36 FetchError c no backend connection
18 FetchError c http first read error: -1 11 (No error recorded)
24 FetchError c http first read error: -1 11 (No error recorded)
22 FetchError c http first read error: -1 11 (No error recorded)
23 FetchError c no backend connection
33 FetchError c no backend connection
25 FetchError c http first read error: -1 11 (No error recorded)
30 FetchError c http first read error: -1 11 (No error recorded)
36 FetchError c no backend connection
17 FetchError c http first read error: -1 11 (No error recorded)
5 FetchError c http first read error: -1 11 (No error recorded)
17 FetchError c http first read error: -1 11 (No error recorded)
34 FetchError c http first read error: -1 11 (No error recorded)
26 FetchError c http first read error: -1 11 (No error recorded)
20 FetchError c http first read error: -1 11 (No error recorded)
5 FetchError c http first read error: -1 11 (No error recorded)
17 FetchError c http first read error: -1 11 (No error recorded)
19 FetchError c http first read error: -1 11 (No error recorded)
7 FetchError c http first read error: -1 11 (No error recorded)
7 FetchError c http first read error: -1 11 (No error recorded)
18 FetchError c http first read error: -1 11 (No error recorded)
7 FetchError c http first read error: -1 11 (No error recorded)
4 FetchError c http first read error: -1 11 (No error recorded)
4 FetchError c http first read error: -1 11 (No error recorded)
7 FetchError c http first read error: -1 11 (No error recorded)
4 FetchError c http first read error: -1 11 (No error recorded)
17 FetchError c http first read error: -1 11 (No error recorded)
20 FetchError c http first read error: -1 11 (No error recorded)
6 FetchError c http first read error: -1 11 (No error recorded)
19 FetchError c http first read error: -1 11 (No error recorded)
4 FetchError c http first read error: -1 11 (No error recorded)
4 FetchError c http first read error: -1 11 (No error recorded)
4 FetchError c http first read error: -1 11 (No error recorded)
4 FetchError c http first read error: -1 11 (No error recorded)
4 FetchError c http first read error: -1 11 (No error recorded)
7 FetchError c http first read error: -1 11 (No error recorded)
7 FetchError c backend write error: 11 (Resource temporarily
unavailable)
4 FetchError c http first read error: -1 11 (No error recorded)
32 FetchError c http first read error: -1 11 (No error recorded)
18 FetchError c no backend connection
7 FetchError c no backend connection
4 FetchError c no backend connection
}}}
I haven't changed the default timeout values for varnish. This is my
configuration for the backends and the director.
{{{
backend radon {
.host = "192.168.3.186";
.port = "80";
.probe = {
.url = "/health-check/";
.interval = 3s;
.window = 5;
.threshold = 2;
}
}
backend xenon {
.host = "192.168.3.187";
.port = "80";
.probe = {
.url = "/health-check/";
.interval = 3s;
.window = 5;
.threshold = 2;
}
}
backend aurum {
.host = "127.0.0.1";
.port = "8085";
.probe = {
.url = "/health-check";
.interval = 3s;
.window = 5;
.threshold = 2;
}
}
backend iridium {
.host = "192.168.3.189";
.port = "8081";
.probe = {
.url = "/varnish-health";
.interval = 3s;
.window = 5;
.threshold = 2;
}
}
director balance client {
{
.backend = radon;
.weight = 1;
}
{
.backend = xenon;
.weight = 1;
}
}
}}}
I have also tried to double the number of threads apache is allowed to run
but without any result. I still get around 10-20 503 error a day and they
come from various requests.
I'm assuming this is some kind of defect/bug. I haven't been able to find
anything about the issues in the Internet. I have set the priority and
severity as a bit higher than normal since out clients receive 503 error
in delicate operations.
Best regards Ronnie
--
Ticket URL: <https://www.varnish-cache.org/trac/ticket/1151>
Varnish <https://varnish-cache.org/>
The Varnish HTTP Accelerator
More information about the varnish-bugs
mailing list