<div dir="ltr">Once you set the backend to sick, you will probably still have requests in-flight to this backend, so you don't want to restart your SSH tunnel just yet. Instead, monitor the VBE.*.BACKENDNAME.conn lines, and wait for them to drop to 0, then you can reset the SSH tunnel.</div><div class="gmail_extra"><br clear="all"><div><div class="gmail_signature" data-smartmail="gmail_signature"><div dir="ltr"><div>-- <br></div>Guillaume Quintard<br></div></div></div>
<br><div class="gmail_quote">On Wed, Nov 15, 2017 at 3:48 PM, Andrei <span dir="ltr"><<a href="mailto:lagged@gmail.com" target="_blank">lagged@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">What do you mean exactly when you say "drain the connections"? :D</div><div class="HOEnZb"><div class="h5"><div class="gmail_extra"><br><div class="gmail_quote">On Wed, Nov 15, 2017 at 8:46 AM, Guillaume Quintard <span dir="ltr"><<a href="mailto:guillaume@varnish-software.com" target="_blank">guillaume@varnish-software.<wbr>com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div class="gmail_extra">Oh, then your life should be easier then! Don't forget to drain the connections, varnishstat will give you the number of open connections open to any backend.</div><div class="gmail_extra"><span class="m_-8398522875067911716HOEnZb"><font color="#888888"><br clear="all"><div><div class="m_-8398522875067911716m_-7945969211251401502gmail_signature" data-smartmail="gmail_signature"><div dir="ltr"><div>-- <br></div>Guillaume Quintard<br></div></div></div></font></span><div><div class="m_-8398522875067911716h5">
<br><div class="gmail_quote">On Wed, Nov 15, 2017 at 3:42 PM, Andrei <span dir="ltr"><<a href="mailto:lagged@gmail.com" target="_blank">lagged@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">Thanks for the pointers! The tunnel setup is pretty flexible so I'll go ahead and mark the backend sick before restarting the tunnel, then healthy once confirmed up.<div><div class="m_-8398522875067911716m_-7945969211251401502h5"><div><br><div class="gmail_extra"><br><div class="gmail_quote">On Wed, Nov 15, 2017 at 8:34 AM, Guillaume Quintard <span dir="ltr"><<a href="mailto:guillaume@varnish-software.com" target="_blank">guillaume@varnish-software.co<wbr>m</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">You can wait until vcl_deliver and do a restart, possibly adding a marker saying "don't bother with the backend, serve from cache".<div><br></div><div>The actual solution would be to mark the backend as sick before restarting the ssh tunnel, and draining the connections, but I guess that's not an option here, is it?</div></div><div class="gmail_extra"><span class="m_-8398522875067911716m_-7945969211251401502m_1579907766627487044HOEnZb"><font color="#888888"><br clear="all"><div><div class="m_-8398522875067911716m_-7945969211251401502m_1579907766627487044m_-2395038880386935429gmail_signature" data-smartmail="gmail_signature"><div dir="ltr"><div>-- <br></div>Guillaume Quintard<br></div></div></div></font></span><div><div class="m_-8398522875067911716m_-7945969211251401502m_1579907766627487044h5">
<br><div class="gmail_quote">On Wed, Nov 15, 2017 at 3:29 PM, Andrei <span dir="ltr"><<a href="mailto:lagged@gmail.com" target="_blank">lagged@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div class="gmail_extra">Hi Guillaume,</div><div class="gmail_extra"><br></div><div class="gmail_extra">Thanks for getting back to me</div><div class="gmail_extra"><br><div class="gmail_quote"><span>On Wed, Nov 15, 2017 at 8:11 AM, Guillaume Quintard <span dir="ltr"><<a href="mailto:guillaume@varnish-software.com" target="_blank">guillaume@varnish-software.co<wbr>m</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">Hi,<div><br></div><div>Why bother with the complex vcl_hit? Since you are saying that the cache is regularly primed, I don't really see the added value.</div></div></blockquote></span><div>I was mainly going by an example a while back, and not all sites/urls are primed in the same manner. It just stuck in the conf ever since</div><span><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div><br></div><div>(note, after a quick glance at it, I think it could just be a race condition where the backend appears up in vcl_hit and is down by the time you ask it the content)</div></div></blockquote></span><div>How would you suggest "restarting" the request to try and force a grace cache object to be returned if present in that case?</div><div><div class="m_-8398522875067911716m_-7945969211251401502m_1579907766627487044m_-2395038880386935429h5"><div><br></div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div class="gmail_extra"><br clear="all"><div><div class="m_-8398522875067911716m_-7945969211251401502m_1579907766627487044m_-2395038880386935429m_-3159864274974919045gmail-m_-2035953708496355986gmail_signature"><div dir="ltr"><div>-- <br></div>Guillaume Quintard<br></div></div></div>
<br><div class="gmail_quote"><div><div class="m_-8398522875067911716m_-7945969211251401502m_1579907766627487044m_-2395038880386935429m_-3159864274974919045gmail-h5">On Wed, Nov 15, 2017 at 6:02 AM, Andrei <span dir="ltr"><<a href="mailto:lagged@gmail.com" target="_blank">lagged@gmail.com</a>></span> wrote:<br></div></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div><div class="m_-8398522875067911716m_-7945969211251401502m_1579907766627487044m_-2395038880386935429m_-3159864274974919045gmail-h5"><div dir="ltr">bump</div><div class="m_-8398522875067911716m_-7945969211251401502m_1579907766627487044m_-2395038880386935429m_-3159864274974919045gmail-m_-2035953708496355986HOEnZb"><div class="m_-8398522875067911716m_-7945969211251401502m_1579907766627487044m_-2395038880386935429m_-3159864274974919045gmail-m_-2035953708496355986h5"><div class="gmail_extra"><br><div class="gmail_quote">On Sun, Nov 5, 2017 at 2:12 AM, Andrei <span dir="ltr"><<a href="mailto:lagged@gmail.com" target="_blank">lagged@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">Hello everyone,<div><br></div><div>One of the backends we have configured, runs through an SSH tunnel which occasionally gets restarted. When the tunnel is restarted, Varnish is returning a 503 since it can't reach the backend for pages which would normally be cached (we force cache on the front page of the related site). I believe our grace implementation might be incorrect, as we would expect a grace period cache return instead of 503.</div><div><br></div><div><div>Our grace ttl is set to 21600 seconds based on a global variable:</div><div><br></div><div>sub vcl_backend_response {</div><div> set beresp.grace = std.duration(variable.global_g<wbr>et("ttl_grace") + "s", 6h);</div><div>}</div><div><br></div><div>Our grace implementation in sub vcl_hit is:</div><div><br></div><div> sub vcl_hit {</div><div> # We have no fresh fish. Lets look at the stale ones.</div><div> if (std.healthy(req.backend_hint)<wbr>) {</div><div> # Backend is healthy. Limit age to 10s.</div><div> if (obj.ttl + 10s > 0s) {</div><div> #set req.http.grace = "normal(limited)";</div><div> std.log("OKHITDELIVER: obj.ttl:" + obj.ttl + " obj.keep: " + obj.keep + " obj.grace: " + obj.grace);</div><div> return (deliver);</div><div> } else {</div><div> # No candidate for grace. Fetch a fresh object.</div><div> std.log("No candidate for grace. Fetch a fresh object. obj.ttl:" + obj.ttl + " obj.keep: " + obj.keep + " obj.grace: " + obj.grace);</div><div> return(miss);</div><div> }</div><div> } else {</div><div> # backend is sick - use full grace</div><div> if (obj.ttl + obj.grace > 0s) {</div><div> #set req.http.grace = "full";</div><div> std.log("SICK DELIVERY: obj.hits: " + obj.hits + " obj.ttl:" + obj.ttl + " obj.keep: " + obj.keep + " obj.grace: " + obj.grace);</div><div> return (deliver);</div><div> } else {</div><div> # no graced object.</div><div> std.log("No graced object. obj.ttl:" + obj.ttl + " obj.keep: " + obj.keep + " obj.grace: " + obj.grace);</div><div> return (miss);</div><div> }</div><div> }</div><div><br></div><div> # fetch & deliver once we get the result</div><div> return (miss); # Dead code, keep as a safeguard</div><div> }</div><div><br></div><div><br></div><div>Occasionally we see:</div><div>- VCL_Log No candidate for grace. Fetch a fresh object. obj.ttl:-1369.659 obj.keep: 0.000 obj.grace: 21600.000</div><div><br></div><div>For the most part, it's:</div><div>- VCL_Log OKHITDELIVER: obj.ttl:26.872 obj.keep: 0.000 obj.grace: 21600.000</div><div><br></div><div>Are we setting the grace ttl too low perhaps?</div></div></div>
</blockquote></div><br></div>
</div></div><br></div></div>______________________________<wbr>_________________<br>
varnish-misc mailing list<br>
<a href="mailto:varnish-misc@varnish-cache.org" target="_blank">varnish-misc@varnish-cache.org</a><br>
<a href="https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc" rel="noreferrer" target="_blank">https://www.varnish-cache.org/<wbr>lists/mailman/listinfo/varnish<wbr>-misc</a><br></blockquote></div><br></div>
</blockquote></div></div></div><br></div></div>
</blockquote></div><br></div></div></div>
</blockquote></div><br></div></div></div></div></div>
</blockquote></div><br></div></div></div></div>
</blockquote></div><br></div>
</div></div></blockquote></div><br></div>