<div dir="ltr">Thanks for sharing!</div><div class="gmail_extra"><br><div class="gmail_quote">On Wed, Oct 18, 2017 at 3:35 PM, Hugues Alary <span dir="ltr"><<a href="mailto:hugues@betabrand.com" target="_blank">hugues@betabrand.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div>Since this could be useful to some other people, here's a bit more details on how it's implemented on my end.<br></div><div><br></div><div>TL;DR: it's automatic. I embedded the script in my docker image and it gets run by k8s when needed.</div><div><br></div><div><div><br></div><div><u>My container image looks like this:</u></div><div><br></div><div>```</div><font face="monospace, monospace">FROM bb:base<br>ENV DEBIAN_FRONTEND noninteractive<br><br>RUN apt-get update<br>RUN apt-get install -y --force-yes varnish<br><br><span style="background-color:rgb(255,217,102)">COPY files/graceful-shutdown.sh /</span><br><br>CMD our-long-varnishd-cmd</font><div>```</div><div><br></div><div><u><font face="monospace, monospace">graceful-shutdown.sh</font> looks like this:</u></div><div><br></div><div>```</div><div><div><font face="monospace, monospace">CONNECTIONS_REMAINING=1</font></div><div><font face="monospace, monospace">while [ $CONNECTIONS_REMAINING != 0 ]; do</font></div><div><font face="monospace, monospace"> CONNECTIONS_REMAINING=`varnish<wbr>stat -1 | awk '/MEMPOOL.sess[0-9]+.live/ {a+=$2} END {print a}'`</font></div><div><font face="monospace, monospace"> echo "$CONNECTIONS_REMAINING remaining, waiting..."</font></div><div><font face="monospace, monospace"> sleep 1</font></div><div><font face="monospace, monospace">done</font></div><div><font face="monospace, monospace">killall varnishd</font></div></div><div>```<br><div><br></div></div><div><u>I then a have a k8s deployment file:</u></div><div><br></div><div>```</div><div><div><font face="monospace, monospace">---</font></div><div><font face="monospace, monospace">apiVersion: extensions/v1beta1</font></div><div><font face="monospace, monospace">kind: Deployment</font></div><div><font face="monospace, monospace">metadata:</font></div><div><font face="monospace, monospace"> name: varnish</font></div><div><font face="monospace, monospace"> labels:</font></div><div><font face="monospace, monospace"> app: varnish</font></div><div><font face="monospace, monospace"> version: "5.2"</font></div><div><font face="monospace, monospace">spec:</font></div><div><font face="monospace, monospace"> replicas: 1</font></div><div><font face="monospace, monospace"> template:</font></div><div><font face="monospace, monospace"> metadata:</font></div><div><font face="monospace, monospace"> labels:</font></div><div><font face="monospace, monospace"> app: varnish</font></div><div><font face="monospace, monospace"> spec:</font></div><div><span style="font-family:monospace,monospace"> terminationGracePeriodSeconds: 300</span><br></div><div><font face="monospace, monospace"> containers:</font></div><div><font face="monospace, monospace"> - name: varnish</font></div><div><font face="monospace, monospace"> image: bb/bb:varnish</font></div><div><font face="monospace, monospace" style="background-color:rgb(241,194,50)"> lifecycle:</font></div><div><font face="monospace, monospace" style="background-color:rgb(241,194,50)"> preStop:</font></div><div><font face="monospace, monospace" style="background-color:rgb(241,194,50)"> exec:</font></div><div><font face="monospace, monospace" style="background-color:rgb(241,194,50)"> command:</font></div><div><font face="monospace, monospace" style="background-color:rgb(241,194,50)"> - "sh"</font></div><div><font face="monospace, monospace" style="background-color:rgb(241,194,50)"> - "/graceful-shutdown.sh"</font></div><div><font face="monospace, monospace"> imagePullPolicy: Always</font></div><div><font face="monospace, monospace"> command: ["</font><span style="font-family:monospace,monospace">our-long-varnishd-cmd"</span><font face="monospace, monospace">]</font></div></div><div>```</div><div><br></div></div><div>The preStop command is run by k8s (when needed**), which will wait until the command exits (up to 300s in this config). Once the preStop command exits, k8s will send a SIGTERM to varnishd (which should already be gone by that time).</div><div><br></div><div>(** `<font face="monospace, monospace">kubectl delete my-varnish-pod</font>` for example, or, if I upload a new deployment file or other conditions triggering a shutdown of the pod.)</div><span class="HOEnZb"><font color="#888888"><div><br></div><div>-H</div></font></span><div><div class="h5"><div class="gmail_extra"><br><div class="gmail_quote">On Wed, Oct 18, 2017 at 1:00 PM, Guillaume Quintard <span dir="ltr"><<a href="mailto:guillaume@varnish-software.com" target="_blank">guillaume@varnish-software.co<wbr>m</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">Glad to hear it! Out of curiosity, do you run that script manually on container you want to kill, or is there a way to do it automatically? </div><div class="gmail_extra"><span class="m_3267110609561703246m_4964548623127318792HOEnZb"><font color="#888888"><br clear="all"><div><div class="m_3267110609561703246m_4964548623127318792m_7060735757829772774gmail_signature" data-smartmail="gmail_signature"><div dir="ltr"><div>-- <br></div>Guillaume Quintard<br></div></div></div></font></span><div><div class="m_3267110609561703246m_4964548623127318792h5">
<br><div class="gmail_quote">On Wed, Oct 18, 2017 at 9:47 PM, Hugues Alary <span dir="ltr"><<a href="mailto:hugues@betabrand.com" target="_blank">hugues@betabrand.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">Your solution works, thanks much.<div><br></div><div>Cheers,</div><div>-Hugues</div></div><div class="m_3267110609561703246m_4964548623127318792m_7060735757829772774HOEnZb"><div class="m_3267110609561703246m_4964548623127318792m_7060735757829772774h5"><div class="gmail_extra"><br><div class="gmail_quote">On Wed, Oct 18, 2017 at 12:21 PM, Hugues Alary <span dir="ltr"><<a href="mailto:hugues@betabrand.com" target="_blank">hugues@betabrand.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><span><div>> <span style="font-size:12.8px">So, there must be a misconception on my, because to me, just refusing new connections isn't graceful. If the LB is sending you new connections, you should honor them.</span><br></div><div><span style="font-size:12.8px"><br></span></div></span><div><span style="font-size:12.8px">You are right. I took some shortcuts in my explanation.</span></div><div><span style="font-size:12.8px"><br></span></div><div><span style="font-size:12.8px">> </span><span style="font-size:12.8px">So, from what I get, k8s stops sending new connections to Varnish, and what you want is is for Varnish to shutdown once there's no more active connections.</span></div><div><span style="font-size:12.8px"><br></span></div><div><span style="font-size:12.8px">That is exactly what happens, indeed. How it happens is irrelevant to our conversation, but, for the sake of being on the same page it's important to assume: k8s stopped sending new connections to the old instance and started sending them to the new instance, and I want old varnish to shutdown once there's no more active connections.</span></div><span><div><span style="font-size:12.8px"><br></span></div><div><span style="font-size:12.8px">> But, let's stop being a pompous jerk obsessed with semantics and let's try to work on a solution.</span><span style="font-size:12.8px"><br></span></div><div><span style="font-size:12.8px"><br></span></div></span><div><span style="font-size:12.8px">All good, it's actually important ;)</span><br></div><div><span style="font-size:12.8px"><br></span></div><div>I will try your solution right away and let you know, in theory it seems like it should work. Thanks for trying to find a solution!</div><span class="m_3267110609561703246m_4964548623127318792m_7060735757829772774m_7294079492193279177HOEnZb"><font color="#888888"><div><br></div><div>-Hugues</div><div><span style="font-size:12.8px"><br></span></div></font></span></div><div class="m_3267110609561703246m_4964548623127318792m_7060735757829772774m_7294079492193279177HOEnZb"><div class="m_3267110609561703246m_4964548623127318792m_7060735757829772774m_7294079492193279177h5"><div class="gmail_extra"><br><div class="gmail_quote">On Wed, Oct 18, 2017 at 12:01 PM, Guillaume Quintard <span dir="ltr"><<a href="mailto:guillaume@varnish-software.com" target="_blank">guillaume@varnish-software.co<wbr>m</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">Hello,<div><br></div><div>So, there must be a misconception on my, because to me, just refusing new connections isn't graceful. If the LB is sending you new connections, you should honor them.</div><div><br></div><div>So, from what I get, k8s stops sending new connections to Varnish, and what you want is is for Varnish to shutdown once there's no more active connections, which is, I would argue, kinda different from refusing new connections. But, let's stop being a pompous jerk obsessed with semantics and let's try to work on a solution.</div><div><br></div><div>From what I get, we can kill varnish as soon as the number of active connections drops to 0, so your apachectl command would be equal to this in Varnish language:</div><div><br></div><blockquote style="margin:0 0 0 40px;border:none;padding:0px"><div><font face="monospace, monospace">if [ `varnishstat -1 | awk '/MEMPOOL.sess[0-9]+.live/ {a+=$2} END {print a}'` != 0 ]; then</font></div><div><font face="monospace, monospace"> sleep 1</font></div><div><font face="monospace, monospace">fi</font></div><div><font face="monospace, monospace">killall varnishd</font></div></blockquote><div><br></div><div>There's probably a shorter version ,but that's the gist of it.</div><div><br></div><div>Would that do?</div></div><div class="gmail_extra"><span class="m_3267110609561703246m_4964548623127318792m_7060735757829772774m_7294079492193279177m_-6588712243730497112HOEnZb"><font color="#888888"><br clear="all"><div><div class="m_3267110609561703246m_4964548623127318792m_7060735757829772774m_7294079492193279177m_-6588712243730497112m_2010082198182511856gmail_signature" data-smartmail="gmail_signature"><div dir="ltr"><div>-- <br></div>Guillaume Quintard<br></div></div></div></font></span><div><div class="m_3267110609561703246m_4964548623127318792m_7060735757829772774m_7294079492193279177m_-6588712243730497112h5">
<br><div class="gmail_quote">On Wed, Oct 18, 2017 at 8:22 PM, Hugues Alary <span dir="ltr"><<a href="mailto:hugues@betabrand.com" target="_blank">hugues@betabrand.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">Hi,<div><br></div><div><div>That is indeed what I want to do. Draining connections at the LB level just involves more work ;).<br></div><div><br></div><div>For context: my entire stack runs on a kubernetes cluster. </div><div><br></div><div>I sometime need to replace a running instance of a server process [be it nginx apache php-fpm varnish whatever] with a new instance. </div><div><br></div><div>Taking Apache as an example, I simply create the new apache instance (a new pod in kubernetes speak), which immediately starts getting some traffic (without changing the load balancer configuration at all, a kubernetes "service" automatically detects new pods), then I gracefully shutdown the old instance (kubernetes actually automatically tells the pod to shutdown), by issuing a "apachectl -k graceful-stop" (kubernetes is configured to issue this command for me), which instructs apache to stop accepting connections, finish, then shutdown.</div><div><br></div><div>It's really great because instead of having to push a new config refusing probes and reload it, I (/kubernetes) simply gracefully stops apache and the traffic flows to the new instance. nginx and php-fpm also handle things this way.</div><div><br></div><div>At any rate, thanks for the advice, I will start using probes!</div><div><br></div><div>Cheers,</div><div>-Hugues</div><div><div class="m_3267110609561703246m_4964548623127318792m_7060735757829772774m_7294079492193279177m_-6588712243730497112m_2010082198182511856h5"><div><br></div><div><br></div><div class="gmail_extra"><br><div class="gmail_quote">On Tue, Oct 17, 2017 at 11:51 PM, Guillaume Quintard <span dir="ltr"><<a href="mailto:guillaume@varnish-software.com" target="_blank">guillaume@varnish-software.co<wbr>m</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="auto">Hi,<div dir="auto"><br></div><div dir="auto">That's not possible. However, what you really want, I think, is not sending new requests to Varnish. That's usually done at the loa-bbalancing level. If your LB use probes, you can tell Varnish to stop honoring them, drain the connections, then kill it.<br><br><div data-smartmail="gmail_signature" dir="auto">-- <br>Guillaume Quintard </div></div></div><div class="gmail_extra"><br><div class="gmail_quote"><div><div class="m_3267110609561703246m_4964548623127318792m_7060735757829772774m_7294079492193279177m_-6588712243730497112m_2010082198182511856m_-6718741731406377904m_8080804183925253129h5">On Oct 18, 2017 02:28, "Hugues Alary" <<a href="mailto:hugues@betabrand.com" target="_blank">hugues@betabrand.com</a>> wrote:<br type="attribution"></div></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div><div class="m_3267110609561703246m_4964548623127318792m_7060735757829772774m_7294079492193279177m_-6588712243730497112m_2010082198182511856m_-6718741731406377904m_8080804183925253129h5"><div dir="ltr">Hi there, <div><br></div><div>I've been looking around and I can't find a documented way of gracefully shutting down varnishd, and by gracefully I mean tell varnish "stop accepting connections, but finish what you were doing, then shutdown".</div><div><br></div><div>I did find something in the "first varnish design notes" (<a href="https://varnish-cache.org/docs/5.1/phk/firstdesign.html" target="_blank">https://varnish-cache.org/doc<wbr>s/5.1/phk/firstdesign.html</a>) which seemed to indicate that sending SIGKILL/SIGTERM would mean "suspend/stop" but KILL doesn't seem to work, and TERM, well... terminates but not gracefully.</div><div><br></div><div>I also tried using "varnishadm stop", which also doesn't gracefully stops connection.</div><div><br></div><div>Is there anyway to achieve this?</div><div><br></div><div>Thanks!</div><div>-Hugues</div></div>
<br></div></div>______________________________<wbr>_________________<br>
varnish-misc mailing list<br>
<a href="mailto:varnish-misc@varnish-cache.org" target="_blank">varnish-misc@varnish-cache.org</a><br>
<a href="https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc" rel="noreferrer" target="_blank">https://www.varnish-cache.org/<wbr>lists/mailman/listinfo/varnish<wbr>-misc</a><br></blockquote></div></div>
</blockquote></div><br></div></div></div></div></div>
</blockquote></div><br></div></div></div>
</blockquote></div><br></div>
</div></div></blockquote></div><br></div>
</div></div></blockquote></div><br></div></div></div>
</blockquote></div><br></div></div></div></div>
<br>______________________________<wbr>_________________<br>
varnish-misc mailing list<br>
<a href="mailto:varnish-misc@varnish-cache.org">varnish-misc@varnish-cache.org</a><br>
<a href="https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc" rel="noreferrer" target="_blank">https://www.varnish-cache.org/<wbr>lists/mailman/listinfo/<wbr>varnish-misc</a><br></blockquote></div><br></div>