Gracefully stopping varnish
Andrei
lagged at gmail.com
Thu Oct 19 05:33:00 UTC 2017
Thanks for sharing!
On Wed, Oct 18, 2017 at 3:35 PM, Hugues Alary <hugues at betabrand.com> wrote:
> Since this could be useful to some other people, here's a bit more details
> on how it's implemented on my end.
>
> TL;DR: it's automatic. I embedded the script in my docker image and it
> gets run by k8s when needed.
>
>
> *My container image looks like this:*
>
> ```
> FROM bb:base
> ENV DEBIAN_FRONTEND noninteractive
>
> RUN apt-get update
> RUN apt-get install -y --force-yes varnish
>
> COPY files/graceful-shutdown.sh /
>
> CMD our-long-varnishd-cmd
> ```
>
> *graceful-shutdown.sh looks like this:*
>
> ```
> CONNECTIONS_REMAINING=1
> while [ $CONNECTIONS_REMAINING != 0 ]; do
> CONNECTIONS_REMAINING=`varnishstat -1 | awk
> '/MEMPOOL.sess[0-9]+.live/ {a+=$2} END {print a}'`
> echo "$CONNECTIONS_REMAINING remaining, waiting..."
> sleep 1
> done
> killall varnishd
> ```
>
> *I then a have a k8s deployment file:*
>
> ```
> ---
> apiVersion: extensions/v1beta1
> kind: Deployment
> metadata:
> name: varnish
> labels:
> app: varnish
> version: "5.2"
> spec:
> replicas: 1
> template:
> metadata:
> labels:
> app: varnish
> spec:
> terminationGracePeriodSeconds: 300
> containers:
> - name: varnish
> image: bb/bb:varnish
> lifecycle:
> preStop:
> exec:
> command:
> - "sh"
> - "/graceful-shutdown.sh"
> imagePullPolicy: Always
> command: ["our-long-varnishd-cmd"]
> ```
>
> The preStop command is run by k8s (when needed**), which will wait until
> the command exits (up to 300s in this config). Once the preStop command
> exits, k8s will send a SIGTERM to varnishd (which should already be gone by
> that time).
>
> (** `kubectl delete my-varnish-pod` for example, or, if I upload a new
> deployment file or other conditions triggering a shutdown of the pod.)
>
> -H
>
> On Wed, Oct 18, 2017 at 1:00 PM, Guillaume Quintard <
> guillaume at varnish-software.com> wrote:
>
>> Glad to hear it! Out of curiosity, do you run that script manually on
>> container you want to kill, or is there a way to do it automatically?
>>
>> --
>> Guillaume Quintard
>>
>> On Wed, Oct 18, 2017 at 9:47 PM, Hugues Alary <hugues at betabrand.com>
>> wrote:
>>
>>> Your solution works, thanks much.
>>>
>>> Cheers,
>>> -Hugues
>>>
>>> On Wed, Oct 18, 2017 at 12:21 PM, Hugues Alary <hugues at betabrand.com>
>>> wrote:
>>>
>>>> > So, there must be a misconception on my, because to me, just
>>>> refusing new connections isn't graceful. If the LB is sending you new
>>>> connections, you should honor them.
>>>>
>>>> You are right. I took some shortcuts in my explanation.
>>>>
>>>> > So, from what I get, k8s stops sending new connections to Varnish,
>>>> and what you want is is for Varnish to shutdown once there's no more active
>>>> connections.
>>>>
>>>> That is exactly what happens, indeed. How it happens is irrelevant to
>>>> our conversation, but, for the sake of being on the same page it's
>>>> important to assume: k8s stopped sending new connections to the old
>>>> instance and started sending them to the new instance, and I want old
>>>> varnish to shutdown once there's no more active connections.
>>>>
>>>> > But, let's stop being a pompous jerk obsessed with semantics and
>>>> let's try to work on a solution.
>>>>
>>>> All good, it's actually important ;)
>>>>
>>>> I will try your solution right away and let you know, in theory it
>>>> seems like it should work. Thanks for trying to find a solution!
>>>>
>>>> -Hugues
>>>>
>>>>
>>>> On Wed, Oct 18, 2017 at 12:01 PM, Guillaume Quintard <
>>>> guillaume at varnish-software.com> wrote:
>>>>
>>>>> Hello,
>>>>>
>>>>> So, there must be a misconception on my, because to me, just refusing
>>>>> new connections isn't graceful. If the LB is sending you new connections,
>>>>> you should honor them.
>>>>>
>>>>> So, from what I get, k8s stops sending new connections to Varnish, and
>>>>> what you want is is for Varnish to shutdown once there's no more active
>>>>> connections, which is, I would argue, kinda different from refusing new
>>>>> connections. But, let's stop being a pompous jerk obsessed with semantics
>>>>> and let's try to work on a solution.
>>>>>
>>>>> From what I get, we can kill varnish as soon as the number of active
>>>>> connections drops to 0, so your apachectl command would be equal to this in
>>>>> Varnish language:
>>>>>
>>>>> if [ `varnishstat -1 | awk '/MEMPOOL.sess[0-9]+.live/ {a+=$2} END
>>>>> {print a}'` != 0 ]; then
>>>>> sleep 1
>>>>> fi
>>>>> killall varnishd
>>>>>
>>>>>
>>>>> There's probably a shorter version ,but that's the gist of it.
>>>>>
>>>>> Would that do?
>>>>>
>>>>> --
>>>>> Guillaume Quintard
>>>>>
>>>>> On Wed, Oct 18, 2017 at 8:22 PM, Hugues Alary <hugues at betabrand.com>
>>>>> wrote:
>>>>>
>>>>>> Hi,
>>>>>>
>>>>>> That is indeed what I want to do. Draining connections at the LB
>>>>>> level just involves more work ;).
>>>>>>
>>>>>> For context: my entire stack runs on a kubernetes cluster.
>>>>>>
>>>>>> I sometime need to replace a running instance of a server process [be
>>>>>> it nginx apache php-fpm varnish whatever] with a new instance.
>>>>>>
>>>>>> Taking Apache as an example, I simply create the new apache instance
>>>>>> (a new pod in kubernetes speak), which immediately starts getting some
>>>>>> traffic (without changing the load balancer configuration at all, a
>>>>>> kubernetes "service" automatically detects new pods), then I gracefully
>>>>>> shutdown the old instance (kubernetes actually automatically tells the pod
>>>>>> to shutdown), by issuing a "apachectl -k graceful-stop" (kubernetes is
>>>>>> configured to issue this command for me), which instructs apache to stop
>>>>>> accepting connections, finish, then shutdown.
>>>>>>
>>>>>> It's really great because instead of having to push a new config
>>>>>> refusing probes and reload it, I (/kubernetes) simply gracefully stops
>>>>>> apache and the traffic flows to the new instance. nginx and php-fpm also
>>>>>> handle things this way.
>>>>>>
>>>>>> At any rate, thanks for the advice, I will start using probes!
>>>>>>
>>>>>> Cheers,
>>>>>> -Hugues
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Tue, Oct 17, 2017 at 11:51 PM, Guillaume Quintard <
>>>>>> guillaume at varnish-software.com> wrote:
>>>>>>
>>>>>>> Hi,
>>>>>>>
>>>>>>> That's not possible. However, what you really want, I think, is not
>>>>>>> sending new requests to Varnish. That's usually done at the loa-bbalancing
>>>>>>> level. If your LB use probes, you can tell Varnish to stop honoring them,
>>>>>>> drain the connections, then kill it.
>>>>>>>
>>>>>>> --
>>>>>>> Guillaume Quintard
>>>>>>>
>>>>>>> On Oct 18, 2017 02:28, "Hugues Alary" <hugues at betabrand.com> wrote:
>>>>>>>
>>>>>>>> Hi there,
>>>>>>>>
>>>>>>>> I've been looking around and I can't find a documented way of
>>>>>>>> gracefully shutting down varnishd, and by gracefully I mean tell varnish
>>>>>>>> "stop accepting connections, but finish what you were doing, then shutdown".
>>>>>>>>
>>>>>>>> I did find something in the "first varnish design notes" (
>>>>>>>> https://varnish-cache.org/docs/5.1/phk/firstdesign.html) which
>>>>>>>> seemed to indicate that sending SIGKILL/SIGTERM would mean "suspend/stop"
>>>>>>>> but KILL doesn't seem to work, and TERM, well... terminates but not
>>>>>>>> gracefully.
>>>>>>>>
>>>>>>>> I also tried using "varnishadm stop", which also doesn't gracefully
>>>>>>>> stops connection.
>>>>>>>>
>>>>>>>> Is there anyway to achieve this?
>>>>>>>>
>>>>>>>> Thanks!
>>>>>>>> -Hugues
>>>>>>>>
>>>>>>>> _______________________________________________
>>>>>>>> varnish-misc mailing list
>>>>>>>> varnish-misc at varnish-cache.org
>>>>>>>> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>
> _______________________________________________
> varnish-misc mailing list
> varnish-misc at varnish-cache.org
> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/varnish-misc/attachments/20171019/1f08aae1/attachment-0001.html>
More information about the varnish-misc
mailing list