<div dir="ltr"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><span style="font-size:12.8000001907349px">that interval and window on your web server is scary..... what you're<br></span><span style="font-size:12.8000001907349px">saying is 'check each web server every 10 minutes, and only fail it<br></span><span style="font-size:12.8000001907349px">after 3 failures'</span></blockquote><div><br></div><div>Hah!! Agreed. I was just trying to rule the connect timeouts out of the picture as to why the failures were happening!</div><div>I plan to set them to more normal intervals once I'm finished testing and I've been able to get this to work.</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><br style="font-size:12.8000001907349px"><span style="font-size:12.8000001907349px">next time you see the issue, look at:</span><br style="font-size:12.8000001907349px"><span style="font-size:12.8000001907349px">varnishadm -n <varnish_name> debug.health</span></blockquote><div><br></div><div>Hmm you may have a point as to the back ends. Varnish is indeed seeing them as 'sick' when I encounter the 503 error:</div><div><br></div><div> </div><div>[root@varnish1:~] #varnishadm -n varnish1 debug.health</div><div>Backend web1 is Sick</div><div>Current states good: 0 threshold: 2 window: 3</div><div>Average responsetime of good probes: 0.000000</div><div>Oldest Newest</div><div>================================================================</div><div>------------------------------------------------------4444444444 Good IPv4</div><div>------------------------------------------------------XXXXXXXXXX Good Xmit</div><div>------------------------------------------------------RRRRRRRRRR Good Recv</div><div>----------------------------------------------------HH---------- Happy</div><div>Backend web2 is Sick</div><div>Current states good: 0 threshold: 2 window: 3</div><div>Average responsetime of good probes: 0.000000</div><div>Oldest Newest</div><div>================================================================</div><div>------------------------------------------------------4444444444 Good IPv4</div><div>------------------------------------------------------XXXXXXXXXX Good Xmit</div><div>------------------------------------------------------RRRRRRRRRR Good Recv</div><div>----------------------------------------------------HH---------- Happy </div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><br style="font-size:12.8000001907349px"><span style="font-size:12.8000001907349px">I'd be willing to bet that varnish is just failing the backends. Try<br></span><span style="font-size:12.8000001907349px">running the healthcheck manually from the varnish boxes:</span><br style="font-size:12.8000001907349px"><span style="font-size:12.8000001907349px">curl -H "Host:</span><a href="http://kiki.example.com/" rel="noreferrer" style="font-size:12.8000001907349px" target="_blank">kiki.example.com</a><span style="font-size:12.8000001907349px">" -v "</span><a href="http://10.10.10.26/healthcheck.php" rel="noreferrer" style="font-size:12.8000001907349px" target="_blank">http://10.10.10.26/healthcheck.php</a><span style="font-size:12.8000001907349px">"</span><br style="font-size:12.8000001907349px"><span style="font-size:12.8000001907349px">And see if you're actually getting good healthchecks. If you're not,<br></span><span style="font-size:12.8000001907349px">then you need to look at your backends (specifically healthcheck.php)</span></blockquote><div><br></div><div>But if I perform the curl you're suggesting, I am able to retrieve the healthcheck.php file!!</div><div><br></div><div>#curl --user admin:somepass -H "Host:<a href="http://wiki.example.com" target="_blank">wiki.example.com</a>" -v "<a href="http://10.10.10.25/healthcheck.php" target="_blank">http://10.10.10.25/healthcheck.php</a>"</div><div>* About to connect() to 52.5.117.61 port 80 (#0)</div><div>* Trying 52.5.117.61... connected</div><div>* Connected to 52.5.117.61 (52.5.117.61) port 80 (#0)</div><div>* Server auth using Basic with user 'admin'</div><div>> GET /healthcheck.php HTTP/1.1</div><div>> Authorization: Basic SomeBase64Hash==</div><div>> User-Agent: curl/7.19.7 (x86_64-redhat-linux-gnu) libcurl/7.19.7 NSS/<a href="http://3.14.0.0" target="_blank">3.14.0.0</a> zlib/1.2.3 libidn/1.18 libssh2/1.4.2</div><div>> Accept: */*</div><div>> Host:<a href="http://wiki.example.com" target="_blank">wiki.example.com</a></div><div>></div><div>< HTTP/1.1 200 OK</div><div>< Date: Thu, 09 Jul 2015 02:10:35 GMT</div><div>< Server: Apache/2.4.6 (CentOS) OpenSSL/1.0.1e-fips mod_fcgid/2.3.9 PHP/5.4.42 SVN/1.7.14 mod_wsgi/3.4 Python/2.7.5</div><div>< X-Powered-By: PHP/5.4.42</div><div>< Content-Length: 5</div><div>< Content-Type: text/html; charset=UTF-8</div><div><</div><div>good</div><div>* Connection #0 to host 52.5.117.61 left intact</div><div>* Closing connection #0 </div><div><br></div><div>But in the curl I just did I was specifying the user auth. Which got me to thinking, maybe I'm handing apache basic auth in the wrong way in my VCL file?</div><div><br></div><div>To test this idea out, I commented out the basic auth lines in my apache config. Then cycled the services on both apache servers and both varnish servers. </div><div><br></div><div>When I ran the test you gave me again, this is the result I got back:</div><div><br></div><div><div>#varnishadm -n varnish1 debug.health</div><div>Backend web1 is Healthy</div><div>Current states good: 3 threshold: 2 window: 3</div><div>Average responsetime of good probes: 0.032781</div><div>Oldest Newest</div><div>================================================================</div><div>---------------------------------------------------------------4 Good IPv4</div><div>---------------------------------------------------------------X Good Xmit</div><div>---------------------------------------------------------------R Good Recv</div><div>-------------------------------------------------------------HHH Happy</div><div>Backend web2 is Healthy</div><div>Current states good: 3 threshold: 2 window: 3</div><div>Average responsetime of good probes: 0.032889</div><div>Oldest Newest</div><div>================================================================</div><div>---------------------------------------------------------------4 Good IPv4</div><div>---------------------------------------------------------------X Good Xmit</div><div>---------------------------------------------------------------R Good Recv</div><div>-------------------------------------------------------------HHH Happy</div></div><div><br></div><div>Everbody's happy again!!</div><div><br></div><div>And I tried browsing around the wiki for quite a long time. And there were NO 503 errors the entire time I was using it. Which tells me that I am, indeed, not handling auth correctly in my VCL.</div><div><br></div><div>The way I thought I solved the problem was by adding a .request to the web server definitions that specified the headers to do a GET on the health check:</div><div><br></div><div><div>.request =</div><div> "GET /healthcheck.php HTTP/1.1"</div><div> "Host: <a href="http://wiki.example.com">wiki.example.com</a>"</div><div> "Connection: close";</div></div><div><br></div><div>The reason I thought this worked was because, after I'd restarted varnish with that change in place I was able to log into the wiki with basic auth in the web browser. And then I'd be able to use it for a while before the back-end would come up as 'sick' in varnish again which would cause the 503 error. </div><div><br></div><div>I then tried following this advice again, which I had also tried earlier without much luck:</div><div><br></div><div><a href="http://blog.tenya.me/blog/2011/12/14/varnish-http-authentication/">http://blog.tenya.me/blog/2011/12/14/varnish-http-authentication/</a><br></div><div><br></div><div>Which tells you to add this section to your VCL file:</div><div><br></div><div><div> if (! req.http.Authorization ~ "Basic SomeBase64Hash==")</div><div> {</div><div> error 401 "Restricted";</div><div> }</div></div><div><br></div><div>And then add this sub_vcl section:</div><div><br></div><div><div>sub vcl_error {</div><div><br></div><div> if (obj.status == 401) {</div><div> set obj.http.Content-Type = "text/html; charset=utf-8";</div><div> set obj.http.WWW-Authenticate = "Basic realm=Secured";</div><div> synthetic {"</div><div><br></div><div> <!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN" "<a href="http://www.w3.org/TR/1999/REC-html401-19991224/loose.dtd">http://www.w3.org/TR/1999/REC-html401-19991224/loose.dtd</a>"></div><div><br></div><div> <HTML></div><div> <HEAD></div><div> <TITLE>Error</TITLE></div><div> <META HTTP-EQUIV='Content-Type' CONTENT='text/html;'></div><div> </HEAD></div><div> <BODY><H1>401 Unauthorized (varnish)</H1></BODY></div><div> </HTML></div><div> "};</div><div> return (deliver);</div><div> }</div><div>}</div></div><div><br></div><div>And after restarting varnish again on both nodes, with authentication in place in the VHOST configs on the web servers I was able to log into the wiki site again and browse around for a while. </div><div><br></div><div>But then after some browsing around the back ends would go sick again and you would see the 503:</div><div><br></div><div><div>#varnishadm -n varnish1 debug.health</div><div>Backend web1 is Sick</div><div>Current states good: 1 threshold: 2 window: 3</div><div>Average responsetime of good probes: 0.000000</div><div>Oldest Newest</div><div>================================================================</div><div>--------------------------------------------------------------44 Good IPv4</div><div>--------------------------------------------------------------XX Good Xmit</div><div>--------------------------------------------------------------RR Good Recv</div><div>------------------------------------------------------------HH-- Happy</div><div>Backend web2 is Sick</div><div>Current states good: 1 threshold: 2 window: 3</div><div>Average responsetime of good probes: 0.000000</div><div>Oldest Newest</div><div>================================================================</div><div>--------------------------------------------------------------44 Good IPv4</div><div>--------------------------------------------------------------XX Good Xmit</div><div>--------------------------------------------------------------RR Good Recv</div><div>------------------------------------------------------------HH-- Happy</div></div><div><br></div><div>So SOMETHING must still be off with how I'm handling authentication in my VCL config. The next step I'm thinking of trying involves passing the authentication headers to the .request section of my web server definition. Although I'm not sure if it'll work. I'll let you guys know if it does. </div><div><br></div><div>But I'd like to present the current state of my VLC again in case anyone has any insight or knowledge to share that may help. </div><div><br></div><div>
<p class="">backend web1 {</p>
<p class=""> .host = "10.10.10.25";</p>
<p class=""> .port = "80";</p>
<p class=""> .connect_timeout = 3600s;</p>
<p class=""> .first_byte_timeout = 3600s;</p>
<p class=""> .between_bytes_timeout = 3600s;</p>
<p class=""> .max_connections = 70;</p>
<p class=""> .probe = {</p>
<p class=""> .request =</p>
<p class=""> "GET /healthcheck.php HTTP/1.1"</p>
<p class=""> "Host: <a href="http://wiki.example.com">wiki.example.com</a>"</p>
<p class=""> "Connection: close";</p>
<p class=""> .interval = 10m;</p>
<p class=""> .timeout = 60s;</p>
<p class=""> .window = 3;</p>
<p class=""> .threshold = 2;</p>
<p class=""> }</p>
<p class="">}</p>
<p class="">backend web2 {</p>
<p class=""> .host = "10.10.10.26";</p>
<p class=""> .port = "80";</p>
<p class=""> .connect_timeout = 3600s;</p>
<p class=""> .first_byte_timeout = 3600s;</p>
<p class=""> .between_bytes_timeout = 3600s;</p>
<p class=""> .max_connections = 70;</p>
<p class=""> .probe = {</p>
<p class=""> .request =</p>
<p class=""> "GET /healthcheck.php HTTP/1.1"</p>
<p class=""> "Host: <a href="http://wiki.example.com">wiki.example.com</a>"</p>
<p class=""> "Connection: close";</p>
<p class=""> .interval = 10m;</p>
<p class=""> .timeout = 60s;</p>
<p class=""> .window = 3;</p>
<p class=""> .threshold = 2;</p>
<p class=""> }</p>
<p class="">}</p>
<p class="">director www round-robin {<br></p>
<p class=""> { .backend = web1; }</p>
<p class=""> { .backend = web2; }</p>
<p class=""> }</p>
<p class="">sub vcl_recv {</p><p class=""> if (! req.http.Authorization ~ "Basic Base64Hash==")<br></p>
<p class=""> {</p>
<p class=""> error 401 "Restricted";</p>
<p class=""> }</p>
<p class=""> if (req.url ~ "&action=submit($|/)") {</p>
<p class=""> return (pass);</p>
<p class=""> }</p>
<p class=""> set req.backend = www;</p>
<p class=""> return (lookup);</p>
<p class="">}</p>
<p class="">sub vcl_fetch {</p>
<p class=""> set beresp.ttl = 3600s;</p>
<p class=""> set beresp.grace = 4h;</p>
<p class=""> return (deliver);</p>
<p class="">}</p>
<p class="">sub vcl_error {</p>
<p class=""> if (obj.status == 401) {</p>
<p class=""> set obj.http.Content-Type = "text/html; charset=utf-8";</p>
<p class=""> set obj.http.WWW-Authenticate = "Basic realm=Secured";</p>
<p class=""> synthetic {"</p>
<p class=""><br></p>
<p class=""> <!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN" "<a href="http://www.w3.org/TR/1999/REC-html401-19991224/loose.dtd">http://www.w3.org/TR/1999/REC-html401-19991224/loose.dtd</a>"></p>
<p class=""><br></p>
<p class=""> <HTML></p>
<p class=""> <HEAD></p>
<p class=""> <TITLE>Error</TITLE></p>
<p class=""> <META HTTP-EQUIV='Content-Type' CONTENT='text/html;'></p>
<p class=""> </HEAD></p>
<p class=""> <BODY><H1>401 Unauthorized (varnish)</H1></BODY></p>
<p class=""> </HTML></p>
<p class=""> "};</p>
<p class=""> return (deliver);</p>
<p class=""> }</p>
<p class="">}</p>
<p class="">sub vcl_deliver {<br></p>
<p class=""> if (obj.hits> 0) {</p>
<p class=""> set resp.http.X-Cache = "HIT";</p>
<p class=""> } else {</p>
<p class=""> set resp.http.X-Cache = "MISS";</p>
<p class=""> }</p>
<p class=""> }</p></div><div>Once again I genuinely appreciate the help of this list, and hope I haven't worn out my welcome! ;)</div><div><br></div><div>Thanks,</div><div>Tim</div><div><br></div></div><div class="gmail_extra"><br><div class="gmail_quote">On Wed, Jul 8, 2015 at 9:31 PM, Jason Price <span dir="ltr"><<a href="mailto:japrice@gmail.com" target="_blank">japrice@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">that interval and window on your web server is scary..... what you're<br>
saying is 'check each web server every 10 minutes, and only fail it<br>
after 3 failures'<br>
<br>
next time you see the issue, look at:<br>
<br>
varnishadm -n <varnish_name> debug.health<br>
<br>
I'd be willing to bet that varnish is just failing the backends. Try<br>
running the healthcheck manually from the varnish boxes:<br>
<br>
curl -H "Host:<a href="http://kiki.example.com" rel="noreferrer" target="_blank">kiki.example.com</a>" -v "<a href="http://10.10.10.26/healthcheck.php" rel="noreferrer" target="_blank">http://10.10.10.26/healthcheck.php</a>"<br>
<br>
And see if you're actually getting good healthchecks. If you're not,<br>
then you need to look at your backends (specifically healthcheck.php)<br>
<div><div class="h5"><br>
On Wed, Jul 8, 2015 at 12:14 PM, Tim Dunphy <<a href="mailto:bluethundr@gmail.com">bluethundr@gmail.com</a>> wrote:<br>
> Hi guys,<br>
><br>
><br>
> I'm having an issue where my varnish server will stop working after a while<br>
> of browsing around the site I'm using it with and throw a 503 server<br>
> unavailable error.<br>
><br>
> In my varnish logs I'm getting a 'no backend connection error':<br>
><br>
> 10 FetchError c no backend connection<br>
> 10 VCL_call c error deliver<br>
> 10 VCL_call c deliver deliver<br>
> 10 TxProtocol c HTTP/1.1<br>
> 10 TxStatus c 503<br>
> 10 TxResponse c Service Unavailable<br>
> 10 TxHeader c Server: Varnish<br>
><br>
><br>
> And if I do a GET on the healthcheck from the command line on the varnish<br>
> server, I get a 503 response from varnish:<br>
><br>
> #GET <a href="http://wiki.example.com/healthcheck.php" rel="noreferrer" target="_blank">http://wiki.example.com/healthcheck.php</a><br>
><br>
> <?xml version="1.0" encoding="utf-8"?><br>
> <!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN"<br>
> "<a href="http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd" rel="noreferrer" target="_blank">http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd</a>"><br>
> <html><br>
> <head><br>
> <title>503 Service Unavailable</title><br>
> </head><br>
> <body><br>
> <h1>Error 503 Service Unavailable</h1><br>
> <p>Service Unavailable</p><br>
> <h3>Guru Meditation:</h3><br>
> <p>XID: <a href="tel:2107225059" value="+12107225059">2107225059</a></p><br>
> <hr><br>
> <p>Varnish cache server</p><br>
> </body><br>
> </html><br>
><br>
> But if I do another GET on the healthcheck file from the varnish server to<br>
> another apache VHOST on the same server as the wiki site that responds to<br>
> the IP of the web server instead of the IP for the varnish server, the GET<br>
> works:<br>
><br>
> #GET <a href="http://ops1.example.com/healthcheck.php" rel="noreferrer" target="_blank">http://ops1.example.com/healthcheck.php</a><br>
> good<br>
><br>
><br>
> So I'm not sure why varnish is having trouble reaching the HC file. The web<br>
> server is a little far from the varnish server. The varnish machines are in<br>
> NYC and the web servers are in northern Virginia.<br>
><br>
> So I tried setting the timeouts in the varnish config to a really high<br>
> number. And that was working for a while. But today I noticed that it<br>
> stopped working. I'll have to restart the varnish service and browse the<br>
> site for a while. Then it'll stop working again and produce the 503 error.<br>
> It's pretty annoying!<br>
><br>
> I was wondering if there might be something in my VCL I could tweak to make<br>
> this work? Or if the fact is that the web servers are simply too far from<br>
> varnish for this to be practical.<br>
><br>
> Here's my VCL file. It's pretty basic:<br>
><br>
> backend web1 {<br>
> .host = "10.10.10.25";<br>
> .port = "80";<br>
> .connect_timeout = 1200s;<br>
> .first_byte_timeout = 1200s;<br>
> .between_bytes_timeout = 1200s;<br>
> .max_connections = 70;<br>
> .probe = {<br>
> .request =<br>
> "GET /healthcheck.php HTTP/1.1"<br>
> "Host: <a href="http://wiki.example.com" rel="noreferrer" target="_blank">wiki.example.com</a>"<br>
> "Connection: close";<br>
> .interval = 10m;<br>
> .timeout = 60s;<br>
> .window = 3;<br>
> .threshold = 2;<br>
> }<br>
> }<br>
><br>
> backend web2 {<br>
> .host = "10.10.10.26";<br>
> .port = "80";<br>
> .connect_timeout = 1200s;<br>
> .first_byte_timeout = 1200s;<br>
> .between_bytes_timeout = 1200s;<br>
> .max_connections = 70;<br>
> .probe = {<br>
> .request =<br>
> "GET /healthcheck.php HTTP/1.1"<br>
> "Host: <a href="http://wiki.example.com" rel="noreferrer" target="_blank">wiki.example.com</a>"<br>
> "Connection: close";<br>
> .interval = 10m;<br>
> .timeout = 60s;<br>
> .window = 3;<br>
> .threshold = 2;<br>
> }<br>
> }<br>
><br>
> director www round-robin {<br>
> { .backend = web1; }<br>
> { .backend = web2; }<br>
> }<br>
><br>
> sub vcl_recv {<br>
><br>
> if (req.url ~ "&action=submit($|/)") {<br>
> return (pass);<br>
> }<br>
><br>
> set req.backend = www;<br>
> return (lookup);<br>
> }<br>
><br>
> sub vcl_fetch {<br>
> set beresp.ttl = 3600s;<br>
> set beresp.grace = 4h;<br>
> return (deliver);<br>
> }<br>
><br>
><br>
> sub vcl_deliver {<br>
> if (obj.hits> 0) {<br>
> set resp.http.X-Cache = "HIT";<br>
> } else {<br>
> set resp.http.X-Cache = "MISS";<br>
> }<br>
> }<br>
><br>
> Thanks,<br>
> Tim<br>
><br>
><br>
><br>
> --<br>
> GPG me!!<br>
><br>
> gpg --keyserver <a href="http://pool.sks-keyservers.net" rel="noreferrer" target="_blank">pool.sks-keyservers.net</a> --recv-keys F186197B<br>
><br>
><br>
</div></div>> _______________________________________________<br>
> varnish-misc mailing list<br>
> <a href="mailto:varnish-misc@varnish-cache.org">varnish-misc@varnish-cache.org</a><br>
> <a href="https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc" rel="noreferrer" target="_blank">https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc</a><br>
</blockquote></div><br><br clear="all"><div><br></div>-- <br><div class="gmail_signature">GPG me!!<br><br>gpg --keyserver <a href="http://pool.sks-keyservers.net" target="_blank">pool.sks-keyservers.net</a> --recv-keys F186197B<br><br></div>
</div>