Breaking Varnish

[Tim Kientzle]

On Jan 28, 2009, at 1:54 AM, Poul-Henning Kamp wrote:
> In message <20090123222947.GB28138 at digdug.corp.631h.metaweb.com>,  
> Niall O'Higgi
> ns writes:
>>>> Can I get you to take -trunk for a spin ?
>>>>
>>>> At least the second of the problems you pasted I'm pretty sure I
>>>> have nailed recently and the first one could easily be the same one
>>>> in a different disguise.
>>
>> I've re-run the load test against varnish-trunk.  Trunk is better
>> behaved, but I now get output like this over and over:
>>
>> child (19731) Started
>> Child (19731) said Closed fds: 4 7 8 10 11
>> Child (19731) said Child starts
>> Child (19731) said managed to mmap 49929912320 bytes of 49929912320
>> Child (19731) said Ready
>> Child (19731) not responding to ping, killing it.
>
> This is a typical indication of raw overload, what levels of traffic
> are you hitting it with ?

Pretty heavy.  We put together a test workload that saturated
Squid at around 1500 req/s on a dual-core dev systems.  The symptoms
above appeared somewhere above 6000 req/s on the same hardware
and workload.

The test has two goals:
  1) To try to find bugs in Varnish that might prevent us from
     switching to Varnish from Squid.
  2) To understand how Varnish behaves when it becomes
     saturated.

When testing Squid in this fashion, we found no bugs.  Under
heavy load, Squid did become very slow but recovered cleanly
and went back into normal operation as soon as the load was
removed.

Varnish didn't fare quite so well.  We did find bugs, as you
know.  Fortunately, those seem to be fixed in trunk.  (When do
you expect the next point release?)

It also appears that Varnish eventually exits completely
if placed under high load.  I'm okay with that as long as it's
intentional behavior; we have a standard nanny that we use in
production to restart crashed services anyway.  Of course,
I understand that killing the child and starting a new one
will also lose the cache, which is obviously not particularly
desirable under heavy load.

Cheers,

Tim