child process of varnishd restarts automatically

[Kristian Lyngstol]

Hi,

On Tue, Dec 21, 2010 at 02:03:17PM -0800, Vincent wrote:
> I think the problem is when varnish uses more than 4GB virtual memory on
> 32-bit, it will restart. I started varnish with  "-s malloc,1.5G". Varnish
> will gradually increase memory usage and stop at about 1.7G. However, the
> virtual memory size is growing continuously and it doesn't seem to stop. We
> restarted varnish yesterday and currently the VM size is 2.7G, and it is
> still growing. Is this normal?

The virtual memory usage depends on a large number of factors, and when you
are limited to 3GB of it (assuming a user/kernel split at 3G/1G)  trying to
determine if there's a leak or not is just not worth the effort. Varnish is
a massively parallel application, each thread has an overhead, each session
has an overhead, each VCL file has an overhead, each object cached has an
overhead. And so forth. These components were all designed for a 64-bit
architecture where address space is a non-issue. Over the years, we've
monitored real memory usage to determine memory leaks and changes in memory
overhead. This effort was, for example, the reason obj_workspace was
removed. Every reported memory leak I've investigated to date has turned
out to be a wild goose chase.

It's taken us quite some time to determine a reasonable estimate for
_real_ memory overhead on 64-bit as it changed from Varnish 2.0 to 2.1, and
I don't think you'll see us try to do the same thing for address space just
because it would be a funny little thing to have on a platform nobody
really need.

If it is of interest, the per-object overhead we are seeing is currently
roughly 1kB for each object - but that's resident memory, not virtual. With
a million objects, that means 1GB of overhead just for the objects -
I wouldn't be surprised if it was twice that in virtual memory.

If you really really need 32-bit for some silly application, spend 100$
buying a dedicated 64-bit-box for Varnish. Because Varnish really really
needs 64-bit. You'll save yourself a great deal of pain and you'll be
running on the platform Varnish was designed for.

> Yes, I know we should have used 32-bit in the first place. The reason is
> that we have some other applications work on 32-bit only. Anyway, we will
> build a 64-bit box dedicated for Varnish. However, isn't it nice to have
> varnish run without any problems on 32-bit machines?

No, we don't care much about 32-bit. Use 64-bit. If you got patches for
32-bit, we'll take a look, but trying to fix a legacy-platform isn't
time well spent. If you fix it, we can check your conclusions, but that's
about it. And in this case, it doesn't look like there's anything to fix.

If I were you, I'd investigate the "32-bit only" application holding you
back, not the 64-bit application actually using your hardware to its
potential. You are trying to tune your Ferrari for a race track in your
back yard: it'll never work well in that scenario and you'll never be happy
with it like that.

I usually call 32-bit platforms "Nintendo platforms" in the context of
Varnish (and compare it to Varnish running on an iPhone, for example): It's
fun if it works, but we wont spend any considerably time tuning it for the
platform. But that's not really a fair thing to say - after all, the
Nintendo 64 came out in 1996.

- Kristian