[Varnish] #1753: Bakend flapping and High load
Varnish
varnish-bugs at varnish-cache.org
Wed Jun 24 19:06:07 CEST 2015
#1753: Bakend flapping and High load
---------------------------------+--------------------
Reporter: zaterio@… | Type: defect
Status: new | Priority: normal
Milestone: Varnish 4.0 release | Component: build
Version: trunk | Severity: normal
Keywords: |
---------------------------------+--------------------
Between 30 min and 2 hours uptime, varnish indicates that one of my
backend is not healthy:
{{{
Backend name Admin Probe
boot.b1 probe Healthy 5/5
boot.vod probe Sick 2/5
boot.b2 probe Healthy 5/5
boot.b3 probe Healthy 5/5
}}}
every 5 segs this backends goes sick in varnishadm, from the first time it
is detected unhealthy CPU load rises, and varnish no longer responds to
curl.
strangely, the backend always respond to curl requests with 200 (curl in
while loop), the vod backend is nginx, and probes (/test) is the status
page.
{{{
backend vod {
.host = "127.0.0.1";
.port = "1000";
.probe = {
.url = "/test";
.interval = 5s;
.timeout = 5s;
.window = 5;
.threshold = 3;
}
}
}}}
{{{
curl 127.0.0.1:1000/test
Active connections: 67
server accepts handled requests
3142415 3142415 3142398
Reading: 0 Writing: 50 Waiting: 17
}}}
{{{
varnish> panic.show
300
Child has not panicked or panic has been cleared
}}}
{{{
varnish> status
200
Child in state running
}}}
{{{
varnishd -V
varnishd (varnish-trunk revision 746b4da)
Copyright (c) 2006 Verdens Gang AS
Copyright (c) 2006-2015 Varnish Software AS
}}}
In varnishstats, VBE was flapping, i capture every 1 seg:
{{{
root at varnish01:/etc/varnish# varnishstat -1|grep VBE|grep happy
VBE.boot.b1.happy 18446744073709551615 . Happy
health probes
VBE.boot.vod.happy 15536286954897674387 . Happy
health probes
VBE.boot.b2.happy 18446744073709551615 . Happy
health probes
VBE.boot.b3.happy 18444492273887459327 . Happy
health probes
root at varnish01:/etc/varnish# varnishstat -1|grep VBE|grep happy
VBE.boot.b1.happy 18446744073709551615 . Happy
health probes
VBE.boot.vod.happy 17845854207609320379 . Happy
health probes
VBE.boot.b2.happy 18446744073709551615 . Happy
health probes
VBE.boot.b3.happy 18446744071427850111 . Happy
health probes
root at varnish01:/etc/varnish# varnishstat -1|grep VBE|grep happy
VBE.boot.b1.happy 18446744073709551615 . Happy
health probes
VBE.boot.vod.happy 12354031126361541413 . Happy
health probes
VBE.boot.b2.happy 18446744073709551615 . Happy
health probes
VBE.boot.b3.happy 18302623353715294207 . Happy
health probes
root at varnish01:/etc/varnish# varnishstat -1|grep VBE|grep happy
VBE.boot.b1.happy 18446744073709551615 . Happy
health probes
VBE.boot.vod.happy 13017343740530059032 . Happy
health probes
VBE.boot.b2.happy 18446744073709551615 . Happy
health probes
VBE.boot.b3.happy 18446708819527426047 . Happy
health probes
root at varnish01:/etc/varnish# varnishstat -1|grep VBE|grep happy
VBE.boot.b1.happy 18446744073709551615 . Happy
health probes
VBE.boot.vod.happy 13359589301309196377 . Happy
health probes
VBE.boot.b2.happy 18446744073709551615 . Happy
health probes
VBE.boot.b3.happy 18446744073709543423 . Happy
health probes
root at varnish01:/etc/varnish# varnishstat -1|grep VBE|grep happy
VBE.boot.b1.happy 18446744073709551615 . Happy
health probes
VBE.boot.vod.happy 3592070478478106612 . Happy
health probes
VBE.boot.b2.happy 18446744073709551615 . Happy
health probes
VBE.boot.b3.happy 18446743935857131519 . Happy
health probes
root at varnish01:/etc/varnish# varnishstat -1|grep VBE|grep happy
VBE.boot.b1.happy 18446744073709551615 . Happy
health probes
VBE.boot.vod.happy 1717656466256125366 . Happy
health probes
VBE.boot.b2.happy 18446744073709551615 . Happy
health probes
VBE.boot.b3.happy 18446603198782177279 . Happy
health probes
root at varnish01:/etc/varnish# varnishstat -1|grep VBE|grep happy
VBE.boot.b1.happy 18446744073709551615 . Happy
health probes
VBE.boot.vod.happy 6961815821132309689 . Happy
health probes
VBE.boot.b2.happy 18446744073709551615 . Happy
health probes
VBE.boot.b3.happy 18446741874686296063 . Happy
health probes
root at varnish01:/etc/varnish# varnishstat -1|grep VBE|grep happy
VBE.boot.b1.happy 18446744073709551615 . Happy
health probes
VBE.boot.vod.happy 15945693309856776194 . Happy
health probes
VBE.boot.b2.happy 18446744073709551615 . Happy
health probes
VBE.boot.b3.happy 0 . Happy health probes
root at varnish01:/etc/varnish# varnishstat -1|grep VBE|grep happy
VBE.boot.b1.happy 18446744073709551615 . Happy
health probes
VBE.boot.vod.happy 14640273067908615597 . Happy
health probes
VBE.boot.b2.happy 18446744073709551615 . Happy
health probes
VBE.boot.b3.happy 18302488130965471231 . Happy
health probes
root at varnish01:/etc/varnish# varnishstat -1|grep VBE|grep happy
VBE.boot.b1.happy 18446744073709551615 . Happy
health probes
VBE.boot.vod.happy 7320422237346507443 . Happy
health probes
VBE.boot.b2.happy 18446744073709551615 . Happy
health probes
VBE.boot.b3.happy 18446744073709518847 . Happy
health probes
root at varnish01:/etc/varnish# varnishstat -1|grep VBE|grep happy
VBE.boot.b1.happy 18446744073709551615 . Happy
health probes
VBE.boot.vod.happy 3193800528665906738 . Happy
health probes
VBE.boot.b2.happy 18446744073709551615 . Happy
health probes
VBE.boot.b3.happy 18443858954913054719 . Happy
health probes
root at varnish01:/etc/varnish# varnishstat -1|grep VBE|grep happy
VBE.boot.b1.happy 18446744073709551615 . Happy
health probes
VBE.boot.vod.happy 15152390158794437337 . Happy
health probes
VBE.boot.b2.happy 18446744073709551615 . Happy
health probes
VBE.boot.b3.happy 18374646605195247615 . Happy
health probes
root at varnish01:/etc/varnish# varnishstat -1|grep VBE|grep happy
VBE.boot.b1.happy 18446744073709551615 . Happy
health probes
VBE.boot.vod.happy 10284485890731071394 . Happy
health probes
VBE.boot.b2.happy 18446744073709551615 . Happy
health probes
VBE.boot.b3.happy 18442081869529743231 . Happy
health probes
root at varnish01:/etc/varnish# varnishstat -1|grep VBE|grep happy
VBE.boot.b1.happy 18446744073709551615 . Happy
health probes
VBE.boot.vod.happy 7147808378940885556 . Happy
health probes
VBE.boot.b2.happy 18446744073709551615 . Happy
health probes
VBE.boot.b3.happy 17870283321137692671 . Happy
health probes
root at varnish01:/etc/varnish# varnishstat -1|grep VBE|grep happy
VBE.boot.b1.happy 18446744073709551615 . Happy
health probes
VBE.boot.vod.happy 16139634276151714427 . Happy
health probes
VBE.boot.b2.happy 18446744073709551615 . Happy
health probes
VBE.boot.b3.happy 18446744065111220223 . Happy
health probes
root at varnish01:/etc/varnish# varnishstat -1|grep VBE|grep happy
VBE.boot.b1.happy 18446744073709551615 . Happy
health probes
VBE.boot.vod.happy 17234149873749130718 . Happy
health probes
VBE.boot.b2.happy 18446744073709551615 . Happy
health probes
VBE.boot.b3.happy 18432107323379407871 . Happy
health probes
root at varnish01:/etc/varnish# varnishstat -1|grep VBE|grep happy
VBE.boot.b1.happy 18446744073709551615 . Happy
health probes
VBE.boot.vod.happy 3761480790024349081 . Happy
health probes
VBE.boot.b2.happy 18446744073709551615 . Happy
health probes
VBE.boot.b3.happy 18302628885096791999 . Happy
health probes
root at varnish01:/etc/varnish# varnishstat -1|grep VBE|grep happy
VBE.boot.b1.happy 18446744073709551615 . Happy
health probes
VBE.boot.vod.happy 14911115919654808265 . Happy
health probes
VBE.boot.b2.happy 18446744073709551615 . Happy
health probes
VBE.boot.b3.happy 18148376200282602879 . Happy
health probes
root at varnish01:/etc/varnish# varnishstat -1|grep VBE|grep happy
VBE.boot.b1.happy 18446744073709551615 . Happy
health probes
VBE.boot.vod.happy 4134780051288534766 . Happy
health probes
VBE.boot.b2.happy 18446744073709551615 . Happy
health probes
VBE.boot.b3.happy 18410703182061633535 . Happy
health probes
}}}
before this machine, with this same configuration, presented #1675, with
the same frecuency.
I upgrade to 746b4da3cafc6146fbcea65dbc741ccf7353d66e in order to fix
#1675
--
Ticket URL: <https://www.varnish-cache.org/trac/ticket/1753>
Varnish <https://varnish-cache.org/>
The Varnish HTTP Accelerator
More information about the varnish-bugs
mailing list