Zero-Downtime Servicing of a Varnish Server Behind a F5 BIG-IP

[Hettwer, Marian]

On 17.07.12 18:45, "Steven Engelhardt" <sengelha at gmail.com> wrote:

>
>
>
>I believe we can achieve a 100% graceful restart if we can get Varnish to
>send a HTTP Connection: Close header for all persistent HTTP connections.
> It would work something like this:
>
>
>1. Mark Varnish server x as out of service in F5
>2. Tell Varnish server x to start closing all persistent HTTP connections
>by sending HTTP Connection: Close headers.  After a HTTP client sees the
>Connection: Close header, it should open up a new HTTP connection for the
>next HTTP request, which
> the F5 will route to the other, in-service Varnish server.
>3. Wait for all HTTP connections to drain from Varnish server x
>4. Service the machine.

This absolutely sounds like a correct plan.
If you achieve doing that, please let me know :)

Cheers,
Marian

>
>
>Steve
>
>On Tue, Jul 17, 2012 at 10:20 AM, Hettwer, Marian
><mhettwer at team.mobile.de> wrote:
>
>Actually the F5 box would have the same problem as you're describing for
>varnish.
>If the F5 box behaves correctly, marking the varnish out of service means
>that the F5 should not accept new connections, but is also not allowed to
>drop existing connections.
>
>Therefor I do wonder how the F5 machine would solve this without errors to
>the client.
>I would bet it also aggressively shuts down existing connections.
>
>Or if it doesn't, well okay, then there is nearly no way around it for
>doing it on the varnish.
>
>
>You might think about installing two varnish boxes in an active/passive
>carp(4) and pfsync(4) setup. But that means you probably have to go with
>FreeBSD on your server. I'm not sure about implementations of carp(4) and
>pfsync(4) for linux.
>
>
>With carp you could do the fail over you wanna do.
>While thinking about it, active/active carp(4) would work too.
>
>So carp(4) / pfsync(4) for the win it is! :-D
>
>HTH,
>Marian
>
>On 09.07.12 23:24, "Steven Engelhardt" <sengelha at gmail.com> wrote:
>
>>Because of the usage patterns of the web service, I'm not sure that even
>>a 1-second timeout would be sufficient to get all HTTP client connections
>>to close.
>>
>>
>>The idea of setting sess_timeout to be 0 was strictly in the context of
>>taking a Varnish server out of service, in order to (somewhat indirectly)
>>force the out of service Varnish server to close all its HTTP
>>connections.  We would restore sess_timeout to a
>> sensible value before putting it back in service.
>>
>>
>>Steve
>>
>>On Mon, Jul 9, 2012 at 4:07 PM, Javier Casares
>><javier at casares.org> wrote:
>>
>>If  sess_timeout  is set by default to 5 seconds, you could try to set
>>this to 4, 3, 2 or 1 and test if has a better performance... i suposse
>>that set this to 0 will close all connectiond and seems to be an epic
>>fail...
>>
>>Javier Casares
>>http://javiercasares.com/
>>
>>
>>2012/7/9 Steven Engelhardt <sengelha at gmail.com>:
>>> Just to be clear, would you suggest something like:
>>> 1. Mark the Varnish server as disabled in the BIG-IP
>>> 2. Use varnishadm to set sess_timeout to 0 to start aggressively
>>>closing
>>> HTTP requests?
>>> 3. Wait for connections to drain
>>>
>>> Steve
>>>
>>> On Mon, Jul 9, 2012 at 3:59 PM, Javier Casares <javier at casares.org>
>>>wrote:
>>>>
>>>>
>>https://www.varnish-cache.org/docs/3.0/reference/varnishd.html
>><https://www.varnish-cache.org/docs/3.0/reference/varnishd.html>
>>>>
>>>> sess_timeout
>>>>
>>>>         Units: seconds
>>>>         Default: 5
>>>>
>>>>     Idle timeout for persistent sessions. If a HTTP request has not
>>>> been received in this many seconds, the session is closed.
>>>>
>>>> ¿?
>>>>
>>>> Javier Casares
>>>> http://javiercasares.com/
>>>>
>>>>
>>>> 2012/7/9 Steven Engelhardt <sengelha at gmail.com>:
>>>> > I have two Varnish servers behind a F5 BIG-IP load balancer.  Both
>>>> > servers
>>>> > are constantly active and serving ~5,000 requests/second 24x7.
>>>>Clients
>>>> > aggressively use HTTP 1.1 Keep-Alives.
>>>> >
>>>> > I'm looking to develop a process for servicing and upgrading the
>>>>Varnish
>>>> > servers which results in 0 downtime to the client.  Our normal
>>>>process
>>>> > for
>>>> > performing changes on production servers is:
>>>> > 1. Mark the server as disabled in the BIG-IP
>>>> > 2. Wait for the connections on the server to drop to 0
>>>> > 3. Service the machine
>>>> >
>>>> > However, because of the aggressive and constant use of HTTP
>>>>Keep-Alives,
>>>> > clients virtually never drop their connections, and the machine
>>>> > continues to
>>>> > serve traffic for quite a long time after it is disabled in the
>>>>BIG-IP.
>>>> >
>>>> > Is there a way to tell Varnish to aggressively close all client HTTP
>>>> > connections?
>>>> >
>>>> > Thank you,
>>>> > Steve
>>>> >
>>>> > --
>>>> > Steven Engelhardt
>>>> > sengelha at gmail.com
>>>> >
>>>> > _______________________________________________
>>>> > varnish-misc mailing list
>>>> > varnish-misc at varnish-cache.org
>>>> >
>>https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
>><https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc>
>>>>
>>>> _______________________________________________
>>>> varnish-misc mailing list
>>>> varnish-misc at varnish-cache.org
>>>>
>>https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
>><https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc>
>>>
>>>
>>>
>>>
>>> --
>>> Steven Engelhardt
>>> sengelha at gmail.com
>>>
>>> _______________________________________________
>>> varnish-misc mailing list
>>> varnish-misc at varnish-cache.org
>>>
>>https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
>><https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc>
>>
>>_______________________________________________
>>varnish-misc mailing list
>>varnish-misc at varnish-cache.org
>>https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
>>
>>
>>
>>
>>
>>
>>
>>
>>--
>>Steven Engelhardt
>>sengelha at gmail.com
>>
>
>
>
>
>
>
>
>
>
>
>-- 
>Steven Engelhardt
>sengelha at gmail.com
>