<html>
<head>
<meta content="text/html; charset=ISO-8859-1"
http-equiv="Content-Type">
</head>
<body bgcolor="#FFFFFF" text="#000000">
Maybe you can start to check the request counters of Apache using
its status module.<br>
<br>
You should be able to see how much requests your webserver is
handling at a given time.<br>
<br>
Check <a class="moz-txt-link-freetext" href="http://httpd.apache.org/docs/2.2/mod/mod_status.html">http://httpd.apache.org/docs/2.2/mod/mod_status.html</a><br>
<br>
It should also be easy to create a script which check every n
seconds the apache status of each servers<br>
and aggregate data for further analysis.<br>
<br>
<div class="moz-cite-prefix">Le 02/06/14 18:08, Tim Dunphy a écrit :<br>
</div>
<blockquote
cite="mid:CAOZy0em-zZYxNXHjYDyk2i9etrMYi-QFjp2SaFbr3W6JP=JoDw@mail.gmail.com"
type="cite">
<div dir="ltr"><br clear="all">
<div>We have two out of six web nodes showing extremely high CPU
usage compared to the others in the farm. </div>
<div><br>
</div>
<div>We're using the round-robin load balancing algorithm. Using
a tool like 'htop' shows us that all the hungry processes
belong to apache. Yet diffing the configuration files between
the nodes with low cpu usage and the ones with high cpu usage
shows no difference at all.</div>
<div><br>
</div>
<div>The theory is that varnish is throwing traffic at these two
hosts too often.</div>
<div><br>
</div>
<div>So I was just wondering if there could be any way to
diagnose this from a Varnish perspective. What clues could
tools like varnishtop and varnishstat or varnishlog yield in
investigating this type of problem?</div>
<div><br>
</div>
<div>Here's my varnish vcl (cleaned of any too-identifying info)
in case anyone can offer an opinion as to why this is going
on:</div>
<div><br>
</div>
<div>
<div>backend web1 {</div>
<div> .host = "10.10.1.98";</div>
<div> .port = "80";</div>
<div> .connect_timeout = 90s;</div>
<div> .first_byte_timeout = 90s;</div>
<div> .between_bytes_timeout = 90s;</div>
<div> .max_connections = 70;</div>
<div> .probe = {</div>
<div> .url = "/healthcheck.php";</div>
<div> .timeout = 5s;</div>
<div> .interval = 30s;</div>
<div> .window = 10;</div>
<div> .threshold = 1;</div>
<div> }</div>
<div>}</div>
<div>
<br>
</div>
<div>backend web2 {</div>
<div> .host = "10.10.1.99";</div>
<div> .port = "80";</div>
<div> .connect_timeout = 90s;</div>
<div> .first_byte_timeout = 90s;</div>
<div> .between_bytes_timeout = 90s;</div>
<div> .max_connections = 70;</div>
<div> .probe = {</div>
<div> .url = "/healthcheck.php";</div>
<div> .timeout = 5s;</div>
<div> .interval = 30s;</div>
<div> .window = 10;</div>
<div>
.threshold = 1;</div>
<div> }</div>
<div>}</div>
<div><br>
</div>
<div>backend web3 {</div>
<div> .host = "10.10.1.100";</div>
<div> .port = "80";</div>
<div> .connect_timeout = 90s;</div>
<div>
.first_byte_timeout = 90s;</div>
<div> .between_bytes_timeout = 90s;</div>
<div> .max_connections = 70;</div>
<div> .probe = {</div>
<div> .url = "/healthcheck.php";</div>
<div> .timeout = 5s;</div>
<div> .interval = 30s;</div>
<div> .window = 10;</div>
<div> .threshold = 1;</div>
<div> }</div>
<div>}</div>
<div><br>
</div>
<div>backend web4 {</div>
<div> .host = "10.10.1.101";</div>
<div>
.port = "80";</div>
<div> .connect_timeout = 90s;</div>
<div> .first_byte_timeout = 90s;</div>
<div> .between_bytes_timeout = 90s;</div>
<div> .max_connections = 70;</div>
<div> .probe = {</div>
<div>
.url = "/healthcheck.php";</div>
<div> .timeout = 5s;</div>
<div> .interval = 30s;</div>
<div> .window = 10;</div>
<div> .threshold = 1;</div>
<div> }</div>
<div>}</div>
<div><br>
</div>
<div>backend web5 {</div>
<div> .host = "10.10.1.235";</div>
<div> .port = "80";</div>
<div> .connect_timeout = 90s;</div>
<div> .first_byte_timeout = 90s;</div>
<div> .between_bytes_timeout = 90s;</div>
<div> .max_connections = 70;</div>
<div> .probe = {</div>
<div> .url = "/healthcheck.php";</div>
<div> .timeout = 5s;</div>
<div> .interval = 30s;</div>
<div> .window = 10;</div>
<div>
.threshold = 1;</div>
<div> }</div>
<div>}</div>
<div><br>
</div>
<div>backend web6 {</div>
<div> .host = "10.10.1.236";</div>
<div> .port = "80";</div>
<div> .connect_timeout = 90s;</div>
<div>
.first_byte_timeout = 90s;</div>
<div> .between_bytes_timeout = 90s;</div>
<div> .max_connections = 70;</div>
<div> .probe = {</div>
<div> .url = "/healthcheck.php";</div>
<div> .timeout = 5s;</div>
<div> .interval = 30s;</div>
<div> .window = 10;</div>
<div> .threshold = 1;</div>
<div> }</div>
<div>}</div>
<div><br>
</div>
<div>backend varnish1 {</div>
<div> .host = "10.10.1.96";</div>
<div>
.port = "80";</div>
<div> .connect_timeout = 90s;</div>
<div> .first_byte_timeout = 90s;</div>
<div> .between_bytes_timeout = 90s;</div>
<div> .max_connections = 1000;</div>
<div>}</div>
<div> </div>
<div>
backend varnish2 {</div>
<div> .host = "10.10.1.97";</div>
<div> .port = "80";</div>
<div> .connect_timeout = 90s;</div>
<div> .first_byte_timeout = 90s;</div>
<div> .between_bytes_timeout = 90s;</div>
<div> .max_connections = 1000;</div>
<div>}</div>
<div><br>
</div>
<div>acl purge {</div>
<div> "localhost";</div>
<div> "127.0.0.1";</div>
<div> "10.10.1.96";</div>
<div> "10.10.1.97";</div>
<div>}</div>
<div> </div>
<div>director www round-robin {</div>
<div> { .backend = web1; }</div>
<div> { .backend = web2; }</div>
<div> { .backend = web3; }</div>
<div> { .backend = web4; }</div>
<div> { .backend = web5; }</div>
<div> { .backend = web6; }</div>
<div><br>
</div>
<div><br>
</div>
<div>}</div>
<div><br>
</div>
<div>director cache round-robin {</div>
<div> { .backend = varnish1; }</div>
<div> { .backend = varnish2; }</div>
<div>}</div>
<div> </div>
<div>sub vcl_recv {</div>
<div> set req.backend = www;</div>
<div> if (req.http.User-Agent ~ "typhoeus") {</div>
<div> error 403 "Temporarily disabled";</div>
<div> }</div>
<div>
set req.grace = 6h;</div>
<div> if (! req.backend.healthy) {</div>
<div> set req.grace = 24h;</div>
<div> }</div>
<div> set req.http.X-Forwarded-For =
req.http.X-Forwarded-For ", " client.ip;</div>
<div> /*if (req.http.host ~ "^origin\.test\.(.+\.|)<a
moz-do-not-send="true" href="http://mywebsite.com">mywebsite.com</a>$")
{</div>
<div> return(pass);</div>
<div> }*/ </div>
<div> if (req.http.host ~
"^origin\.(.+\.|)mywebsite\.com$") {</div>
<div> return(pass);</div>
<div> }</div>
<div> if (req.http.host ~ "<a moz-do-not-send="true"
href="http://test.cms.mywebsite.com">test.cms.mywebsite.com</a>|<a
moz-do-not-send="true"
href="http://test.cms-2.mywebsite.com">test.cms-2.mywebsite.com</a>|<a
moz-do-not-send="true"
href="http://test.facebook.mywebsite.com">test.facebook.mywebsite.com</a>")
{</div>
<div> return (pass);</div>
<div> }</div>
<div> if (req.http.host ~ "<a moz-do-not-send="true"
href="http://test.cache-only.mywebsite.com">test.cache-only.mywebsite.com</a>")
{</div>
<div> set req.backend = cache;</div>
<div> }</div>
<div> /* if (req.http.host ~ "test\.*\.<a
moz-do-not-send="true" href="http://mywebsite.com">mywebsite.com</a>")
{</div>
<div> if (req.http.host !~
"test\.(.+\.|)m\.mywebsite\.com$") {</div>
<div> set req.http.host = "<a moz-do-not-send="true"
href="http://test.mywebsite.com">test.mywebsite.com</a>";</div>
<div> }</div>
<div> } else {</div>
<div> return (pass);</div>
<div> }*/</div>
<div><br>
</div>
<div> if (req.http.host ~ ".*\.<a moz-do-not-send="true"
href="http://mywebsite.com">mywebsite.com</a>") {</div>
<div> if (req.http.host !~ "(.+\.|)m\.mywebsite\.com$") {</div>
<div> set req.http.host = "<a moz-do-not-send="true"
href="http://www.mywebsite.com">www.mywebsite.com</a>";</div>
<div> }</div>
<div> } else {</div>
<div> return (pass);</div>
<div> }</div>
<div> </div>
<div>/*</div>
<div> if (req.url ~
"^/node/75402/tweettracker_brand/config/xml") {</div>
<div> error 403 "Temporarily disabled";</div>
<div> }</div>
<div>*/</div>
<div> if (req.request == "PURGE") {</div>
<div> if (!client.ip ~ purge) {</div>
<div> error 405 "Not allowed.";</div>
<div> }</div>
<div> return (lookup);</div>
<div> }</div>
<div> if (req.request != "GET" &&</div>
<div> req.request != "HEAD" &&</div>
<div> req.request != "PUT" &&</div>
<div> req.request != "POST" &&</div>
<div> req.request != "TRACE" &&</div>
<div> req.request != "OPTIONS" &&</div>
<div> req.request != "DELETE") {</div>
<div> return (pipe);</div>
<div> }</div>
<div> if (req.request != "GET" && req.request !=
"HEAD") {</div>
<div> return (pass);</div>
<div> }</div>
<div> if (req.url ~ "eyeblaster/addineyeV2.html\?.*") {</div>
<div> set req.url = "/eyeblaster/addineyeV2.html";</div>
<div> }</div>
<div> if (req.url ~
"sites/all/modules/custom/bravo_ad/ads.html\?.*") {</div>
<div> set req.url =
"/sites/all/modules/custom/bravo_ad/ads.html";</div>
<div> }</div>
<div> if (req.url ~ "sites/<a moz-do-not-send="true"
href="http://m.mywebsite.com/modules/custom/bravo_ad/ads.html%5C?.*">m.mywebsite.com/modules/custom/bravo_ad/ads.html\?.*</a>")
{</div>
<div> set req.url = "/sites/<a moz-do-not-send="true"
href="http://m.mywebsite.com/modules/custom/bravo_ad/ads.html">m.mywebsite.com/modules/custom/bravo_ad/ads.html</a>";</div>
<div> }</div>
<div> if (req.url ~
"ahah_helper\.php|bravo_points\.php|install\.php|update\.php|cron\.php|/json(:?\?.*)?$")
{</div>
<div> return (pass);</div>
<div> }</div>
<div> if (req.http.Authorization) {</div>
<div> return (pass);</div>
<div> }</div>
<div> if (req.url ~ "login" || req.url ~ "logout") {</div>
<div> return (pass);</div>
<div> }</div>
<div> if (req.url ~ "^/admin/" || req.url ~ "^/node/add/")
{</div>
<div> return (pass);</div>
<div> }</div>
<div> if (req.http.Cache-Control ~ "no-cache") {</div>
<div> //return (pass);</div>
<div> }</div>
<div> if (req.http.Cookie ~
"(VARNISH|DRUPAL_UID|LOGGED_IN|SESS|_twitter_sess)") {</div>
<div> set req.http.Cookie = regsuball(req.http.Cookie,
"(^|;\s*)(__[a-z]+|has_js)=[^;]*", "");</div>
<div> set req.http.Cookie = regsub(req.http.Cookie,
"^;\s*", "");</div>
<div> } else {</div>
<div> unset req.http.Cookie;</div>
<div> }</div>
<div> if (req.restarts == 0) {</div>
<div>
if (client.ip == "10.10.1.96" || client.ip ==
"10.10.1.97") {</div>
<div> set req.backend = www;</div>
<div> } elsif (server.ip == "10.10.1.96") {</div>
<div> set req.backend = varnish2;</div>
<div> } else {</div>
<div> set req.backend = varnish1;</div>
<div> }</div>
<div> } elsif (req.restarts >= 2) {</div>
<div> return (pass);</div>
<div> }</div>
<div> if (req.restarts >= 2) {</div>
<div> return (pass);</div>
<div> }</div>
<div> if (req.url ~
"\.(ico|jpg|jpeg|png|gif|gz|tgz|bz2|tbz|mp3|ogg|swf|ICO|JPG|JPEG|PNG|GIF|GZ|TGZ|BZ2|TBZ|MP3|OOG|SWF)")
{</div>
<div> unset req.http.Accept-Encoding;</div>
<div> }</div>
<div> if (req.url ~
"^/(sites/all/modules/mywebsite_admanager/includes/ads.php|doubleclick/DARTIframe.html)(\?.*|)$")
{</div>
<div> set req.url = regsub(req.url, "\?.*$", "");</div>
<div> }</div>
<div>/* keep user-agents</div>
<div> if (! req.http.User-Agent ~ "MSIE") {</div>
<div> set req.http.User-Agent = "Mozilla";</div>
<div> }</div>
<div>*/</div>
<div> if (req.http.Accept-Encoding ~ "gzip") {</div>
<div> set req.http.Accept-Encoding = "gzip";</div>
<div> } elsif (req.http.Accept-Encoding ~ "deflate") {</div>
<div> set req.http.Accept-Encoding = "deflate";</div>
<div> } else {</div>
<div> unset req.http.Accept-Encoding;</div>
<div> }</div>
<div> return (lookup);</div>
<div>}</div>
<div> </div>
<div>sub vcl_pipe {</div>
<div> set bereq.http.connection = "close";</div>
<div> return (pipe);</div>
<div>}</div>
<div> </div>
<div>sub vcl_pass {</div>
<div> return (pass);</div>
<div>}</div>
<div> </div>
<div>sub vcl_hash {</div>
<div> set req.hash += req.url;</div>
<div> set req.hash += req.http.host;</div>
<div> if (req.http.Cookie ~ "VARNISH|DRUPAL_UID|LOGGED_IN")
{</div>
<div> set req.hash += req.http.Cookie;</div>
<div> }</div>
<div> return (hash);</div>
<div>}</div>
<div><br>
</div>
<div>sub vcl_hit {</div>
<div> if (req.request == "PURGE") {</div>
<div> set obj.ttl = 0s;</div>
<div> error 200 "Purged.";</div>
<div> }</div>
<div> /*</div>
<div> if (!obj.cacheable) {</div>
<div> return (pass);</div>
<div> }</div>
<div> return (deliver);</div>
<div> */</div>
<div>
}</div>
<div> </div>
<div>sub vcl_fetch {</div>
<div> if (beresp.status == 500) {</div>
<div> set req.http.X-Varnish-Error = "1";</div>
<div> restart;</div>
<div> }</div>
<div> set beresp.grace = 6h;</div>
<div> # Set a short circuit cache lifetime for resp codes
above 302</div>
<div> if (beresp.status > 302) {</div>
<div><span class="" style="white-space:pre"> </span>set
beresp.ttl = 60s;</div>
<div><span class="" style="white-space:pre"> </span>set
beresp.http.Cache-Control = "max-age = 60";</div>
<div> }</div>
<div> if (beresp.http.Edge-control ~ "no-store") {</div>
<div> set beresp.http.storage = "1";</div>
<div> set beresp.cacheable = false;</div>
<div> return (pass);</div>
<div> }</div>
<div> if (beresp.status >= 300 || !beresp.cacheable) {</div>
<div> set beresp.http.Varnish-X-Cacheable = "Not
Cacheable";</div>
<div> set beresp.http.storage = "1";</div>
<div> return (pass);</div>
<div> }</div>
<div> if (beresp.http.Set-Cookie) {</div>
<div> return (pass);</div>
<div> }</div>
<div> if (beresp.cacheable) {</div>
<div> unset beresp.http.expires;</div>
<div> set beresp.ttl = 600s;</div>
<div> set beresp.http.Cache-Control = "max-age = 600";</div>
<div> if (req.url ~
"\.(ico|jpg|jpeg|png|gif|gz|tgz|bz2|tbz|mp3|ogg|swf|ICO|JPG|JPEG|PNG|GIF|GZ|TGZ|BZ2|TBZ|MP3|OOG|SWF)")
{</div>
<div> set beresp.ttl = 43829m;</div>
<div> set beresp.http.Cache-Control = "max-age =
1000000";</div>
<div> }</div>
<div> /*</div>
<div> if (req.url ~ "json(:?\??.*)?$") {</div>
<div> set beresp.ttl = 30s;</div>
<div> set beresp.http.Cache-Control = "max-age =
30";</div>
<div> }</div>
<div> */</div>
<div> }</div>
<div> return (deliver);</div>
<div>}</div>
<div> </div>
<div>sub vcl_deliver {</div>
<div> if (obj.hits > 0) {</div>
<div> set resp.http.Varnish-X-Cache = "HIT";</div>
<div> set resp.http.Varnish-X-Cache-Hits = obj.hits;</div>
<div> } else {</div>
<div> set resp.http.Varnish-X-Cache = "MISS";</div>
<div> }</div>
<div> return (deliver);</div>
<div>}</div>
<div> </div>
<div>sub vcl_error {</div>
<div> if (req.restarts == 0) {</div>
<div> return (restart);</div>
<div> }</div>
<div> if (req.http.X-Varnish-Error != "1") {</div>
<div> set req.http.X-Varnish-Error = "1";</div>
<div> return (restart);</div>
<div> }</div>
<div> set obj.http.Content-Type = "text/html;
charset=utf-8";</div>
<div> synthetic {"</div>
<div> <?xml version="1.0" encoding="utf-8"?></div>
<div> <!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0
Strict//EN"</div>
<div> "<a moz-do-not-send="true"
href="http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd</a>"
<<a moz-do-not-send="true"
href="http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd%22">http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd%22</a>>
></div>
<div><html></div>
<div> <head></div>
<div> <title><a moz-do-not-send="true"
href="http://mywebsite.com">mywebsite.com</a> "}
obj.status " " obj.response {"</title></div>
<div> <style type="text/css"></div>
<div> body {background-color: #fff; margin: auto;}</div>
<div> #page {width: 1000px; margin: 0 auto;
background-color: #fff;}</div>
<div> </style></div>
<div> </head></div>
<div> <body></div>
<div> <div id="page"></div>
<div> <img
src="<a moz-do-not-send="true"
href="http://www.mywebsite.com/media/site-unavailable/site-unavailable.jpg">http://www.mywebsite.com/media/site-unavailable/site-unavailable.jpg</a>"
width="1000" height="700" ></div>
<div> <!--</div>
<div> <h4>Debug Info:</h4></div>
<div> <pre></div>
<div> Status: "} obj.status {"</div>
<div>
Response: "} obj.response {"</div>
<div> XID: "} req.xid {"</div>
<div> </pre></div>
<div> <address><a href="<a
moz-do-not-send="true"
href="http://www.varnish-cache.org/">http://www.varnish-cache.org/</a>">Varnish</a></address<<a
moz-do-not-send="true"
href="http://www.varnish-cache.org/%22%3eVarnish%3c/a%3e%3c/address">http://www.varnish-cache.org/%22%3eVarnish%3c/a%3e%3c/address</a>>
<<a moz-do-not-send="true"
href="http://www.varnish-cache.org/%22%3eVarnish%3c/a%3e%3c/address">http://www.varnish-cache.org/%22%3eVarnish%3c/a%3e%3c/address</a>>
></div>
<div> --></div>
<div> </div></div>
<div> </body></div>
<div></html></div>
<div> "};</div>
<div> return (deliver);</div>
<div>}</div>
</div>
<div><br>
</div>
<div><br>
</div>
<div>Thanks</div>
<div>Tim</div>
-- <br>
GPG me!!<br>
<br>
gpg --keyserver <a moz-do-not-send="true"
href="http://pool.sks-keyservers.net" target="_blank">pool.sks-keyservers.net</a>
--recv-keys F186197B<br>
<br>
</div>
<br>
<fieldset class="mimeAttachmentHeader"></fieldset>
<br>
<pre wrap="">_______________________________________________
varnish-misc mailing list
<a class="moz-txt-link-abbreviated" href="mailto:varnish-misc@varnish-cache.org">varnish-misc@varnish-cache.org</a>
<a class="moz-txt-link-freetext" href="https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc">https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc</a></pre>
</blockquote>
<br>
<pre class="moz-signature" cols="72">--
Boris Guéry
twitter: @borisguery
skype: borisguery
mobile: +33 6 86 83 03 12
pgp: 0x034C6265
irc: bgy @ freenode</pre>
</body>
</html>