<div dir="auto"><div>I'm afraid I'll have to skip that step since the tunnel is restarted based on connection errors from probes sent every 3s, and those "pending requests" shouldn't be completing correctly anyways as those are half open connections at that point. Not to mention I want to minimize visibility/downtime, and waiting on the lingering requests also means more timeouts to worry about. Thanks again for your input though. I should definitely see better results from request tagging/restarts and marking the backends accordingly during the tunnel reatarts<br><div class="gmail_extra"><br><div class="gmail_quote">On Nov 15, 2017 16:56, "Guillaume Quintard" <<a href="mailto:guillaume@varnish-software.com">guillaume@varnish-software.com</a>> wrote:<br type="attribution"><blockquote class="quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">Once you set the backend to sick, you will probably still have requests in-flight to this backend, so you don't want to restart your SSH tunnel just yet. Instead, monitor the VBE.*.BACKENDNAME.conn lines, and wait for them to drop to 0, then you can reset the SSH tunnel.</div><div class="gmail_extra"><font color="#888888"><br clear="all"><div><div class="m_-6449959077462274477gmail_signature" data-smartmail="gmail_signature"><div dir="ltr"><div>-- <br></div>Guillaume Quintard<br></div></div></div></font><div class="elided-text">
<br><div class="gmail_quote">On Wed, Nov 15, 2017 at 3:48 PM, Andrei <span dir="ltr"><<a href="mailto:lagged@gmail.com" target="_blank">lagged@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">What do you mean exactly when you say "drain the connections"? :D</div><div class="m_-6449959077462274477HOEnZb"><div class="m_-6449959077462274477h5"><div class="gmail_extra"><br><div class="gmail_quote">On Wed, Nov 15, 2017 at 8:46 AM, Guillaume Quintard <span dir="ltr"><<a href="mailto:guillaume@varnish-software.com" target="_blank">guillaume@varnish-software.co<wbr>m</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div class="gmail_extra">Oh, then your life should be easier then! Don't forget to drain the connections, varnishstat will give you the number of open connections open to any backend.</div><div class="gmail_extra"><span class="m_-6449959077462274477m_-8398522875067911716HOEnZb"><font color="#888888"><br clear="all"><div><div class="m_-6449959077462274477m_-8398522875067911716m_-7945969211251401502gmail_signature" data-smartmail="gmail_signature"><div dir="ltr"><div>-- <br></div>Guillaume Quintard<br></div></div></div></font></span><div><div class="m_-6449959077462274477m_-8398522875067911716h5">
<br><div class="gmail_quote">On Wed, Nov 15, 2017 at 3:42 PM, Andrei <span dir="ltr"><<a href="mailto:lagged@gmail.com" target="_blank">lagged@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">Thanks for the pointers! The tunnel setup is pretty flexible so I'll go ahead and mark the backend sick before restarting the tunnel, then healthy once confirmed up.<div><div class="m_-6449959077462274477m_-8398522875067911716m_-7945969211251401502h5"><div><br><div class="gmail_extra"><br><div class="gmail_quote">On Wed, Nov 15, 2017 at 8:34 AM, Guillaume Quintard <span dir="ltr"><<a href="mailto:guillaume@varnish-software.com" target="_blank">guillaume@varnish-software.co<wbr>m</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">You can wait until vcl_deliver and do a restart, possibly adding a marker saying "don't bother with the backend, serve from cache".<div><br></div><div>The actual solution would be to mark the backend as sick before restarting the ssh tunnel, and draining the connections, but I guess that's not an option here, is it?</div></div><div class="gmail_extra"><span class="m_-6449959077462274477m_-8398522875067911716m_-7945969211251401502m_1579907766627487044HOEnZb"><font color="#888888"><br clear="all"><div><div class="m_-6449959077462274477m_-8398522875067911716m_-7945969211251401502m_1579907766627487044m_-2395038880386935429gmail_signature" data-smartmail="gmail_signature"><div dir="ltr"><div>-- <br></div>Guillaume Quintard<br></div></div></div></font></span><div><div class="m_-6449959077462274477m_-8398522875067911716m_-7945969211251401502m_1579907766627487044h5">
<br><div class="gmail_quote">On Wed, Nov 15, 2017 at 3:29 PM, Andrei <span dir="ltr"><<a href="mailto:lagged@gmail.com" target="_blank">lagged@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div class="gmail_extra">Hi Guillaume,</div><div class="gmail_extra"><br></div><div class="gmail_extra">Thanks for getting back to me</div><div class="gmail_extra"><br><div class="gmail_quote"><span>On Wed, Nov 15, 2017 at 8:11 AM, Guillaume Quintard <span dir="ltr"><<a href="mailto:guillaume@varnish-software.com" target="_blank">guillaume@varnish-software.co<wbr>m</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">Hi,<div><br></div><div>Why bother with the complex vcl_hit? Since you are saying that the cache is regularly primed, I don't really see the added value.</div></div></blockquote></span><div>I was mainly going by an example a while back, and not all sites/urls are primed in the same manner. It just stuck in the conf ever since</div><span><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div><br></div><div>(note, after a quick glance at it, I think it could just be a race condition where the backend appears up in vcl_hit and is down by the time you ask it the content)</div></div></blockquote></span><div>How would you suggest "restarting" the request to try and force a grace cache object to be returned if present in that case?</div><div><div class="m_-6449959077462274477m_-8398522875067911716m_-7945969211251401502m_1579907766627487044m_-2395038880386935429h5"><div><br></div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div class="gmail_extra"><br clear="all"><div><div class="m_-6449959077462274477m_-8398522875067911716m_-7945969211251401502m_1579907766627487044m_-2395038880386935429m_-3159864274974919045gmail-m_-2035953708496355986gmail_signature"><div dir="ltr"><div>-- <br></div>Guillaume Quintard<br></div></div></div>
<br><div class="gmail_quote"><div><div class="m_-6449959077462274477m_-8398522875067911716m_-7945969211251401502m_1579907766627487044m_-2395038880386935429m_-3159864274974919045gmail-h5">On Wed, Nov 15, 2017 at 6:02 AM, Andrei <span dir="ltr"><<a href="mailto:lagged@gmail.com" target="_blank">lagged@gmail.com</a>></span> wrote:<br></div></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div><div class="m_-6449959077462274477m_-8398522875067911716m_-7945969211251401502m_1579907766627487044m_-2395038880386935429m_-3159864274974919045gmail-h5"><div dir="ltr">bump</div><div class="m_-6449959077462274477m_-8398522875067911716m_-7945969211251401502m_1579907766627487044m_-2395038880386935429m_-3159864274974919045gmail-m_-2035953708496355986HOEnZb"><div class="m_-6449959077462274477m_-8398522875067911716m_-7945969211251401502m_1579907766627487044m_-2395038880386935429m_-3159864274974919045gmail-m_-2035953708496355986h5"><div class="gmail_extra"><br><div class="gmail_quote">On Sun, Nov 5, 2017 at 2:12 AM, Andrei <span dir="ltr"><<a href="mailto:lagged@gmail.com" target="_blank">lagged@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">Hello everyone,<div><br></div><div>One of the backends we have configured, runs through an SSH tunnel which occasionally gets restarted. When the tunnel is restarted, Varnish is returning a 503 since it can't reach the backend for pages which would normally be cached (we force cache on the front page of the related site). I believe our grace implementation might be incorrect, as we would expect a grace period cache return instead of 503.</div><div><br></div><div><div>Our grace ttl is set to 21600 seconds based on a global variable:</div><div><br></div><div>sub vcl_backend_response {</div><div> set beresp.grace = std.duration(variable.global_g<wbr>et("ttl_grace") + "s", 6h);</div><div>}</div><div><br></div><div>Our grace implementation in sub vcl_hit is:</div><div><br></div><div> sub vcl_hit {</div><div> # We have no fresh fish. Lets look at the stale ones.</div><div> if (std.healthy(req.backend_hint)<wbr>) {</div><div> # Backend is healthy. Limit age to 10s.</div><div> if (obj.ttl + 10s > 0s) {</div><div> #set req.http.grace = "normal(limited)";</div><div> std.log("OKHITDELIVER: obj.ttl:" + obj.ttl + " obj.keep: " + obj.keep + " obj.grace: " + obj.grace);</div><div> return (deliver);</div><div> } else {</div><div> # No candidate for grace. Fetch a fresh object.</div><div> std.log("No candidate for grace. Fetch a fresh object. obj.ttl:" + obj.ttl + " obj.keep: " + obj.keep + " obj.grace: " + obj.grace);</div><div> return(miss);</div><div> }</div><div> } else {</div><div> # backend is sick - use full grace</div><div> if (obj.ttl + obj.grace > 0s) {</div><div> #set req.http.grace = "full";</div><div> std.log("SICK DELIVERY: obj.hits: " + obj.hits + " obj.ttl:" + obj.ttl + " obj.keep: " + obj.keep + " obj.grace: " + obj.grace);</div><div> return (deliver);</div><div> } else {</div><div> # no graced object.</div><div> std.log("No graced object. obj.ttl:" + obj.ttl + " obj.keep: " + obj.keep + " obj.grace: " + obj.grace);</div><div> return (miss);</div><div> }</div><div> }</div><div><br></div><div> # fetch & deliver once we get the result</div><div> return (miss); # Dead code, keep as a safeguard</div><div> }</div><div><br></div><div><br></div><div>Occasionally we see:</div><div>- VCL_Log No candidate for grace. Fetch a fresh object. obj.ttl:-1369.659 obj.keep: 0.000 obj.grace: 21600.000</div><div><br></div><div>For the most part, it's:</div><div>- VCL_Log OKHITDELIVER: obj.ttl:26.872 obj.keep: 0.000 obj.grace: 21600.000</div><div><br></div><div>Are we setting the grace ttl too low perhaps?</div></div></div>
</blockquote></div><br></div>
</div></div><br></div></div>______________________________<wbr>_________________<br>
varnish-misc mailing list<br>
<a href="mailto:varnish-misc@varnish-cache.org" target="_blank">varnish-misc@varnish-cache.org</a><br>
<a href="https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc" rel="noreferrer" target="_blank">https://www.varnish-cache.org/<wbr>lists/mailman/listinfo/varnish<wbr>-misc</a><br></blockquote></div><br></div>
</blockquote></div></div></div><br></div></div>
</blockquote></div><br></div></div></div>
</blockquote></div><br></div></div></div></div></div>
</blockquote></div><br></div></div></div></div>
</blockquote></div><br></div>
</div></div></blockquote></div><br></div></div>
</blockquote></div><br></div></div></div>