Zero-Downtime Servicing of a Varnish Server Behind a F5 BIG-IP

[Steven Engelhardt]

I believe we can achieve a 100% graceful restart if we can get Varnish to
send a HTTP Connection: Close header for all persistent HTTP connections.
 It would work something like this:

1. Mark Varnish server *x* as out of service in F5
2. Tell Varnish server *x* to start closing all persistent HTTP connections
by sending HTTP Connection: Close headers.  After a HTTP client sees the
Connection: Close header, it should open up a new HTTP connection for the
next HTTP request, which the F5 will route to the other, in-service Varnish
server.
3. Wait for all HTTP connections to drain from Varnish server *x*
4. Service the machine.

Steve

On Tue, Jul 17, 2012 at 10:20 AM, Hettwer, Marian
<mhettwer at team.mobile.de>wrote:

> Actually the F5 box would have the same problem as you're describing for
> varnish.
> If the F5 box behaves correctly, marking the varnish out of service means
> that the F5 should not accept new connections, but is also not allowed to
> drop existing connections.
>
> Therefor I do wonder how the F5 machine would solve this without errors to
> the client.
> I would bet it also aggressively shuts down existing connections.
>
> Or if it doesn't, well okay, then there is nearly no way around it for
> doing it on the varnish.
>
>
> You might think about installing two varnish boxes in an active/passive
> carp(4) and pfsync(4) setup. But that means you probably have to go with
> FreeBSD on your server. I'm not sure about implementations of carp(4) and
> pfsync(4) for linux.
>
>
> With carp you could do the fail over you wanna do.
> While thinking about it, active/active carp(4) would work too.
>
> So carp(4) / pfsync(4) for the win it is! :-D
>
> HTH,
> Marian
>
> On 09.07.12 23:24, "Steven Engelhardt" <sengelha at gmail.com> wrote:
>
> >Because of the usage patterns of the web service, I'm not sure that even
> >a 1-second timeout would be sufficient to get all HTTP client connections
> >to close.
> >
> >
> >The idea of setting sess_timeout to be 0 was strictly in the context of
> >taking a Varnish server out of service, in order to (somewhat indirectly)
> >force the out of service Varnish server to close all its HTTP
> >connections.  We would restore sess_timeout to a
> > sensible value before putting it back in service.
> >
> >
> >Steve
> >
> >On Mon, Jul 9, 2012 at 4:07 PM, Javier Casares
> ><javier at casares.org> wrote:
> >
> >If  sess_timeout  is set by default to 5 seconds, you could try to set
> >this to 4, 3, 2 or 1 and test if has a better performance... i suposse
> >that set this to 0 will close all connectiond and seems to be an epic
> >fail...
> >
> >Javier Casares
> >http://javiercasares.com/
> >
> >
> >2012/7/9 Steven Engelhardt <sengelha at gmail.com>:
> >> Just to be clear, would you suggest something like:
> >> 1. Mark the Varnish server as disabled in the BIG-IP
> >> 2. Use varnishadm to set sess_timeout to 0 to start aggressively closing
> >> HTTP requests?
> >> 3. Wait for connections to drain
> >>
> >> Steve
> >>
> >> On Mon, Jul 9, 2012 at 3:59 PM, Javier Casares <javier at casares.org>
> >>wrote:
> >>>
> >>>
> >https://www.varnish-cache.org/docs/3.0/reference/varnishd.html
> ><https://www.varnish-cache.org/docs/3.0/reference/varnishd.html>
> >>>
> >>> sess_timeout
> >>>
> >>>         Units: seconds
> >>>         Default: 5
> >>>
> >>>     Idle timeout for persistent sessions. If a HTTP request has not
> >>> been received in this many seconds, the session is closed.
> >>>
> >>> ¿?
> >>>
> >>> Javier Casares
> >>> http://javiercasares.com/
> >>>
> >>>
> >>> 2012/7/9 Steven Engelhardt <sengelha at gmail.com>:
> >>> > I have two Varnish servers behind a F5 BIG-IP load balancer.  Both
> >>> > servers
> >>> > are constantly active and serving ~5,000 requests/second 24x7.
> >>>Clients
> >>> > aggressively use HTTP 1.1 Keep-Alives.
> >>> >
> >>> > I'm looking to develop a process for servicing and upgrading the
> >>>Varnish
> >>> > servers which results in 0 downtime to the client.  Our normal
> >>>process
> >>> > for
> >>> > performing changes on production servers is:
> >>> > 1. Mark the server as disabled in the BIG-IP
> >>> > 2. Wait for the connections on the server to drop to 0
> >>> > 3. Service the machine
> >>> >
> >>> > However, because of the aggressive and constant use of HTTP
> >>>Keep-Alives,
> >>> > clients virtually never drop their connections, and the machine
> >>> > continues to
> >>> > serve traffic for quite a long time after it is disabled in the
> >>>BIG-IP.
> >>> >
> >>> > Is there a way to tell Varnish to aggressively close all client HTTP
> >>> > connections?
> >>> >
> >>> > Thank you,
> >>> > Steve
> >>> >
> >>> > --
> >>> > Steven Engelhardt
> >>> > sengelha at gmail.com
> >>> >
> >>> > _______________________________________________
> >>> > varnish-misc mailing list
> >>> > varnish-misc at varnish-cache.org
> >>> >
> >https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
> ><https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc>
> >>>
> >>> _______________________________________________
> >>> varnish-misc mailing list
> >>> varnish-misc at varnish-cache.org
> >>>
> >https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
> ><https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc>
> >>
> >>
> >>
> >>
> >> --
> >> Steven Engelhardt
> >> sengelha at gmail.com
> >>
> >> _______________________________________________
> >> varnish-misc mailing list
> >> varnish-misc at varnish-cache.org
> >>
> >https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
> ><https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc>
> >
> >_______________________________________________
> >varnish-misc mailing list
> >varnish-misc at varnish-cache.org
> >https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
> >
> >
> >
> >
> >
> >
> >
> >
> >--
> >Steven Engelhardt
> >sengelha at gmail.com
> >
>
>


-- 
Steven Engelhardt
sengelha at gmail.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/varnish-misc/attachments/20120717/a1410371/attachment-0001.html>