Gracefully stopping varnish
Hugues Alary
hugues at betabrand.com
Wed Oct 18 20:35:05 UTC 2017
Since this could be useful to some other people, here's a bit more details
on how it's implemented on my end.
TL;DR: it's automatic. I embedded the script in my docker image and it gets
run by k8s when needed.
*My container image looks like this:*
```
FROM bb:base
ENV DEBIAN_FRONTEND noninteractive
RUN apt-get update
RUN apt-get install -y --force-yes varnish
COPY files/graceful-shutdown.sh /
CMD our-long-varnishd-cmd
```
*graceful-shutdown.sh looks like this:*
```
CONNECTIONS_REMAINING=1
while [ $CONNECTIONS_REMAINING != 0 ]; do
CONNECTIONS_REMAINING=`varnishstat -1 | awk '/MEMPOOL.sess[0-9]+.live/
{a+=$2} END {print a}'`
echo "$CONNECTIONS_REMAINING remaining, waiting..."
sleep 1
done
killall varnishd
```
*I then a have a k8s deployment file:*
```
---
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
name: varnish
labels:
app: varnish
version: "5.2"
spec:
replicas: 1
template:
metadata:
labels:
app: varnish
spec:
terminationGracePeriodSeconds: 300
containers:
- name: varnish
image: bb/bb:varnish
lifecycle:
preStop:
exec:
command:
- "sh"
- "/graceful-shutdown.sh"
imagePullPolicy: Always
command: ["our-long-varnishd-cmd"]
```
The preStop command is run by k8s (when needed**), which will wait until
the command exits (up to 300s in this config). Once the preStop command
exits, k8s will send a SIGTERM to varnishd (which should already be gone by
that time).
(** `kubectl delete my-varnish-pod` for example, or, if I upload a new
deployment file or other conditions triggering a shutdown of the pod.)
-H
On Wed, Oct 18, 2017 at 1:00 PM, Guillaume Quintard <
guillaume at varnish-software.com> wrote:
> Glad to hear it! Out of curiosity, do you run that script manually on
> container you want to kill, or is there a way to do it automatically?
>
> --
> Guillaume Quintard
>
> On Wed, Oct 18, 2017 at 9:47 PM, Hugues Alary <hugues at betabrand.com>
> wrote:
>
>> Your solution works, thanks much.
>>
>> Cheers,
>> -Hugues
>>
>> On Wed, Oct 18, 2017 at 12:21 PM, Hugues Alary <hugues at betabrand.com>
>> wrote:
>>
>>> > So, there must be a misconception on my, because to me, just refusing
>>> new connections isn't graceful. If the LB is sending you new connections,
>>> you should honor them.
>>>
>>> You are right. I took some shortcuts in my explanation.
>>>
>>> > So, from what I get, k8s stops sending new connections to Varnish,
>>> and what you want is is for Varnish to shutdown once there's no more active
>>> connections.
>>>
>>> That is exactly what happens, indeed. How it happens is irrelevant to
>>> our conversation, but, for the sake of being on the same page it's
>>> important to assume: k8s stopped sending new connections to the old
>>> instance and started sending them to the new instance, and I want old
>>> varnish to shutdown once there's no more active connections.
>>>
>>> > But, let's stop being a pompous jerk obsessed with semantics and let's
>>> try to work on a solution.
>>>
>>> All good, it's actually important ;)
>>>
>>> I will try your solution right away and let you know, in theory it seems
>>> like it should work. Thanks for trying to find a solution!
>>>
>>> -Hugues
>>>
>>>
>>> On Wed, Oct 18, 2017 at 12:01 PM, Guillaume Quintard <
>>> guillaume at varnish-software.com> wrote:
>>>
>>>> Hello,
>>>>
>>>> So, there must be a misconception on my, because to me, just refusing
>>>> new connections isn't graceful. If the LB is sending you new connections,
>>>> you should honor them.
>>>>
>>>> So, from what I get, k8s stops sending new connections to Varnish, and
>>>> what you want is is for Varnish to shutdown once there's no more active
>>>> connections, which is, I would argue, kinda different from refusing new
>>>> connections. But, let's stop being a pompous jerk obsessed with semantics
>>>> and let's try to work on a solution.
>>>>
>>>> From what I get, we can kill varnish as soon as the number of active
>>>> connections drops to 0, so your apachectl command would be equal to this in
>>>> Varnish language:
>>>>
>>>> if [ `varnishstat -1 | awk '/MEMPOOL.sess[0-9]+.live/ {a+=$2} END
>>>> {print a}'` != 0 ]; then
>>>> sleep 1
>>>> fi
>>>> killall varnishd
>>>>
>>>>
>>>> There's probably a shorter version ,but that's the gist of it.
>>>>
>>>> Would that do?
>>>>
>>>> --
>>>> Guillaume Quintard
>>>>
>>>> On Wed, Oct 18, 2017 at 8:22 PM, Hugues Alary <hugues at betabrand.com>
>>>> wrote:
>>>>
>>>>> Hi,
>>>>>
>>>>> That is indeed what I want to do. Draining connections at the LB level
>>>>> just involves more work ;).
>>>>>
>>>>> For context: my entire stack runs on a kubernetes cluster.
>>>>>
>>>>> I sometime need to replace a running instance of a server process [be
>>>>> it nginx apache php-fpm varnish whatever] with a new instance.
>>>>>
>>>>> Taking Apache as an example, I simply create the new apache instance
>>>>> (a new pod in kubernetes speak), which immediately starts getting some
>>>>> traffic (without changing the load balancer configuration at all, a
>>>>> kubernetes "service" automatically detects new pods), then I gracefully
>>>>> shutdown the old instance (kubernetes actually automatically tells the pod
>>>>> to shutdown), by issuing a "apachectl -k graceful-stop" (kubernetes is
>>>>> configured to issue this command for me), which instructs apache to stop
>>>>> accepting connections, finish, then shutdown.
>>>>>
>>>>> It's really great because instead of having to push a new config
>>>>> refusing probes and reload it, I (/kubernetes) simply gracefully stops
>>>>> apache and the traffic flows to the new instance. nginx and php-fpm also
>>>>> handle things this way.
>>>>>
>>>>> At any rate, thanks for the advice, I will start using probes!
>>>>>
>>>>> Cheers,
>>>>> -Hugues
>>>>>
>>>>>
>>>>>
>>>>> On Tue, Oct 17, 2017 at 11:51 PM, Guillaume Quintard <
>>>>> guillaume at varnish-software.com> wrote:
>>>>>
>>>>>> Hi,
>>>>>>
>>>>>> That's not possible. However, what you really want, I think, is not
>>>>>> sending new requests to Varnish. That's usually done at the loa-bbalancing
>>>>>> level. If your LB use probes, you can tell Varnish to stop honoring them,
>>>>>> drain the connections, then kill it.
>>>>>>
>>>>>> --
>>>>>> Guillaume Quintard
>>>>>>
>>>>>> On Oct 18, 2017 02:28, "Hugues Alary" <hugues at betabrand.com> wrote:
>>>>>>
>>>>>>> Hi there,
>>>>>>>
>>>>>>> I've been looking around and I can't find a documented way of
>>>>>>> gracefully shutting down varnishd, and by gracefully I mean tell varnish
>>>>>>> "stop accepting connections, but finish what you were doing, then shutdown".
>>>>>>>
>>>>>>> I did find something in the "first varnish design notes" (
>>>>>>> https://varnish-cache.org/docs/5.1/phk/firstdesign.html) which
>>>>>>> seemed to indicate that sending SIGKILL/SIGTERM would mean "suspend/stop"
>>>>>>> but KILL doesn't seem to work, and TERM, well... terminates but not
>>>>>>> gracefully.
>>>>>>>
>>>>>>> I also tried using "varnishadm stop", which also doesn't gracefully
>>>>>>> stops connection.
>>>>>>>
>>>>>>> Is there anyway to achieve this?
>>>>>>>
>>>>>>> Thanks!
>>>>>>> -Hugues
>>>>>>>
>>>>>>> _______________________________________________
>>>>>>> varnish-misc mailing list
>>>>>>> varnish-misc at varnish-cache.org
>>>>>>> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/varnish-misc/attachments/20171018/9815fc1f/attachment.html>
More information about the varnish-misc
mailing list