Varnish virtual memory usage

[Henry Paulissen]

We encounter the same problem.

Its seems to occur only on x64 platforms.
We decided to take a different approach and installed vmware to the machine.
Next we did a setup of 6 guests with x32 PAE software.

No strange memory leaks occurred since then at the price of small storage (3.5G max) and limited worker threads (256 max).

Opened a ticket for the problem, but the wont listen until I buy a support contract (á €8K).
Seems they don’t want to know there is some kind of memory issue in their software.

Anyway...
Varnish is running stable now with some few tricks.


Regards,

-----Original Message-----
From: varnish-misc-bounces at projects.linpro.no [mailto:varnish-misc-bounces at projects.linpro.no] On Behalf Of Kristian Lyngstol
Sent: woensdag 21 oktober 2009 13:34
To: Roi Avinoam
Cc: varnish-misc at projects.linpro.no
Subject: Re: Varnish virtual memory usage

On Mon, Sep 21, 2009 at 02:55:07PM +0300, Roi Avinoam wrote:
> At Metacafe we're testing the integration with Varnish, and I was 
> tasked with benchmarking our Varnish setup. I intentionally 
> over-flooded the server with requests, in an attempt to see how the 
> system will behave under extensive traffic. Surprisingly, the server 
> ran out of swap and crashed.

That seems mighty strange. What sort of tests did you do?

> In out configuration, "-s file,/var/lib/varnish/varnish_storage.bin,1G".
> Does it mean Varnish shouldn't use more than 1GB of the virtual memory?
> Is there any other way to limit the memory/storage usage?

If you are using -s file and you have 4GB of memory, you are telling Varnish to create a _file_ of 1GB, and it's up to the kernel what it keeps in memory or not. If you actually run out of memory with this setup, you've either hit a bug (need more details first), or you're doing something strange like having the mmaped file (/var/lib/varnish/) in tmpfs with a sizelimit less than 1GB or something along those lines. But I need more details to say anything for certain.

--
Kristian Lyngstøl
Redpill Linpro AS
Tlf: +47 21544179
Mob: +47 99014497