[Varnish] #1753: Bakend flapping and High load

Varnish varnish-bugs at varnish-cache.org
Wed Jun 24 19:06:07 CEST 2015


#1753: Bakend flapping and High load
---------------------------------+--------------------
 Reporter:  zaterio@…            |       Type:  defect
   Status:  new                  |   Priority:  normal
Milestone:  Varnish 4.0 release  |  Component:  build
  Version:  trunk                |   Severity:  normal
 Keywords:                       |
---------------------------------+--------------------
 Between 30 min and 2 hours uptime, varnish  indicates that one of my
 backend is not healthy:

 {{{
 Backend name                   Admin      Probe
 boot.b1                      probe      Healthy 5/5
 boot.vod                       probe      Sick 2/5
 boot.b2                   probe      Healthy 5/5
 boot.b3                   probe      Healthy 5/5
 }}}

 every 5 segs this backends goes sick in varnishadm, from the first time it
 is detected unhealthy CPU load rises, and varnish no longer responds to
 curl.
 strangely, the backend always respond to curl requests with 200 (curl in
 while loop), the vod backend is nginx, and probes (/test) is the status
 page.

 {{{
 backend vod {
    .host = "127.0.0.1";
    .port = "1000";
    .probe = {
       .url = "/test";
       .interval = 5s;
       .timeout = 5s;
       .window = 5;
       .threshold = 3;
    }
 }
 }}}

 {{{
 curl 127.0.0.1:1000/test
 Active connections: 67
 server accepts handled requests
  3142415 3142415 3142398
 Reading: 0 Writing: 50 Waiting: 17
 }}}


 {{{
 varnish> panic.show
 300
 Child has not panicked or panic has been cleared
 }}}

 {{{
 varnish> status
 200
 Child in state running
 }}}


 {{{
 varnishd -V
 varnishd (varnish-trunk revision 746b4da)
 Copyright (c) 2006 Verdens Gang AS
 Copyright (c) 2006-2015 Varnish Software AS
 }}}

 In varnishstats, VBE was flapping, i capture every 1 seg:

 {{{
 root at varnish01:/etc/varnish# varnishstat -1|grep VBE|grep happy
 VBE.boot.b1.happy               18446744073709551615          .   Happy
 health probes
 VBE.boot.vod.happy                15536286954897674387          .   Happy
 health probes
 VBE.boot.b2.happy            18446744073709551615          .   Happy
 health probes
 VBE.boot.b3.happy            18444492273887459327          .   Happy
 health probes
 root at varnish01:/etc/varnish# varnishstat -1|grep VBE|grep happy
 VBE.boot.b1.happy               18446744073709551615          .   Happy
 health probes
 VBE.boot.vod.happy                17845854207609320379          .   Happy
 health probes
 VBE.boot.b2.happy            18446744073709551615          .   Happy
 health probes
 VBE.boot.b3.happy            18446744071427850111          .   Happy
 health probes
 root at varnish01:/etc/varnish# varnishstat -1|grep VBE|grep happy
 VBE.boot.b1.happy               18446744073709551615          .   Happy
 health probes
 VBE.boot.vod.happy                12354031126361541413          .   Happy
 health probes
 VBE.boot.b2.happy            18446744073709551615          .   Happy
 health probes
 VBE.boot.b3.happy            18302623353715294207          .   Happy
 health probes
 root at varnish01:/etc/varnish# varnishstat -1|grep VBE|grep happy
 VBE.boot.b1.happy               18446744073709551615          .   Happy
 health probes
 VBE.boot.vod.happy                13017343740530059032          .   Happy
 health probes
 VBE.boot.b2.happy            18446744073709551615          .   Happy
 health probes
 VBE.boot.b3.happy            18446708819527426047          .   Happy
 health probes
 root at varnish01:/etc/varnish# varnishstat -1|grep VBE|grep happy
 VBE.boot.b1.happy               18446744073709551615          .   Happy
 health probes
 VBE.boot.vod.happy                13359589301309196377          .   Happy
 health probes
 VBE.boot.b2.happy            18446744073709551615          .   Happy
 health probes
 VBE.boot.b3.happy            18446744073709543423          .   Happy
 health probes
 root at varnish01:/etc/varnish# varnishstat -1|grep VBE|grep happy
 VBE.boot.b1.happy               18446744073709551615          .   Happy
 health probes
 VBE.boot.vod.happy                3592070478478106612          .   Happy
 health probes
 VBE.boot.b2.happy            18446744073709551615          .   Happy
 health probes
 VBE.boot.b3.happy            18446743935857131519          .   Happy
 health probes
 root at varnish01:/etc/varnish# varnishstat -1|grep VBE|grep happy
 VBE.boot.b1.happy               18446744073709551615          .   Happy
 health probes
 VBE.boot.vod.happy                1717656466256125366          .   Happy
 health probes
 VBE.boot.b2.happy            18446744073709551615          .   Happy
 health probes
 VBE.boot.b3.happy            18446603198782177279          .   Happy
 health probes
 root at varnish01:/etc/varnish# varnishstat -1|grep VBE|grep happy
 VBE.boot.b1.happy               18446744073709551615          .   Happy
 health probes
 VBE.boot.vod.happy                6961815821132309689          .   Happy
 health probes
 VBE.boot.b2.happy            18446744073709551615          .   Happy
 health probes
 VBE.boot.b3.happy            18446741874686296063          .   Happy
 health probes
 root at varnish01:/etc/varnish# varnishstat -1|grep VBE|grep happy
 VBE.boot.b1.happy               18446744073709551615          .   Happy
 health probes
 VBE.boot.vod.happy                15945693309856776194          .   Happy
 health probes
 VBE.boot.b2.happy            18446744073709551615          .   Happy
 health probes
 VBE.boot.b3.happy                       0          .   Happy health probes
 root at varnish01:/etc/varnish# varnishstat -1|grep VBE|grep happy
 VBE.boot.b1.happy               18446744073709551615          .   Happy
 health probes
 VBE.boot.vod.happy                14640273067908615597          .   Happy
 health probes
 VBE.boot.b2.happy            18446744073709551615          .   Happy
 health probes
 VBE.boot.b3.happy            18302488130965471231          .   Happy
 health probes
 root at varnish01:/etc/varnish# varnishstat -1|grep VBE|grep happy
 VBE.boot.b1.happy               18446744073709551615          .   Happy
 health probes
 VBE.boot.vod.happy                7320422237346507443          .   Happy
 health probes
 VBE.boot.b2.happy            18446744073709551615          .   Happy
 health probes
 VBE.boot.b3.happy            18446744073709518847          .   Happy
 health probes
 root at varnish01:/etc/varnish# varnishstat -1|grep VBE|grep happy
 VBE.boot.b1.happy               18446744073709551615          .   Happy
 health probes
 VBE.boot.vod.happy                3193800528665906738          .   Happy
 health probes
 VBE.boot.b2.happy            18446744073709551615          .   Happy
 health probes
 VBE.boot.b3.happy            18443858954913054719          .   Happy
 health probes
 root at varnish01:/etc/varnish# varnishstat -1|grep VBE|grep happy
 VBE.boot.b1.happy               18446744073709551615          .   Happy
 health probes
 VBE.boot.vod.happy                15152390158794437337          .   Happy
 health probes
 VBE.boot.b2.happy            18446744073709551615          .   Happy
 health probes
 VBE.boot.b3.happy            18374646605195247615          .   Happy
 health probes
 root at varnish01:/etc/varnish# varnishstat -1|grep VBE|grep happy
 VBE.boot.b1.happy               18446744073709551615          .   Happy
 health probes
 VBE.boot.vod.happy                10284485890731071394          .   Happy
 health probes
 VBE.boot.b2.happy            18446744073709551615          .   Happy
 health probes
 VBE.boot.b3.happy            18442081869529743231          .   Happy
 health probes
 root at varnish01:/etc/varnish# varnishstat -1|grep VBE|grep happy
 VBE.boot.b1.happy               18446744073709551615          .   Happy
 health probes
 VBE.boot.vod.happy                7147808378940885556          .   Happy
 health probes
 VBE.boot.b2.happy            18446744073709551615          .   Happy
 health probes
 VBE.boot.b3.happy            17870283321137692671          .   Happy
 health probes
 root at varnish01:/etc/varnish# varnishstat -1|grep VBE|grep happy
 VBE.boot.b1.happy               18446744073709551615          .   Happy
 health probes
 VBE.boot.vod.happy                16139634276151714427          .   Happy
 health probes
 VBE.boot.b2.happy            18446744073709551615          .   Happy
 health probes
 VBE.boot.b3.happy            18446744065111220223          .   Happy
 health probes
 root at varnish01:/etc/varnish# varnishstat -1|grep VBE|grep happy
 VBE.boot.b1.happy               18446744073709551615          .   Happy
 health probes
 VBE.boot.vod.happy                17234149873749130718          .   Happy
 health probes
 VBE.boot.b2.happy            18446744073709551615          .   Happy
 health probes
 VBE.boot.b3.happy            18432107323379407871          .   Happy
 health probes
 root at varnish01:/etc/varnish# varnishstat -1|grep VBE|grep happy
 VBE.boot.b1.happy               18446744073709551615          .   Happy
 health probes
 VBE.boot.vod.happy                3761480790024349081          .   Happy
 health probes
 VBE.boot.b2.happy            18446744073709551615          .   Happy
 health probes
 VBE.boot.b3.happy            18302628885096791999          .   Happy
 health probes
 root at varnish01:/etc/varnish# varnishstat -1|grep VBE|grep happy
 VBE.boot.b1.happy               18446744073709551615          .   Happy
 health probes
 VBE.boot.vod.happy                14911115919654808265          .   Happy
 health probes
 VBE.boot.b2.happy            18446744073709551615          .   Happy
 health probes
 VBE.boot.b3.happy            18148376200282602879          .   Happy
 health probes
 root at varnish01:/etc/varnish# varnishstat -1|grep VBE|grep happy
 VBE.boot.b1.happy               18446744073709551615          .   Happy
 health probes
 VBE.boot.vod.happy                4134780051288534766          .   Happy
 health probes
 VBE.boot.b2.happy            18446744073709551615          .   Happy
 health probes
 VBE.boot.b3.happy            18410703182061633535          .   Happy
 health probes
 }}}

 before this machine, with this same configuration, presented #1675, with
 the same frecuency.
 I upgrade to 746b4da3cafc6146fbcea65dbc741ccf7353d66e in order to fix
 #1675

-- 
Ticket URL: <https://www.varnish-cache.org/trac/ticket/1753>
Varnish <https://varnish-cache.org/>
The Varnish HTTP Accelerator



More information about the varnish-bugs mailing list