Varnish virtual memory usage

[Roi Avinoam]

Thanks for your reply.

1. What I did was create 100 simultaneous processes, and each process requested the same page (with 'curl'):
	a. Once with the exact same URL - which resulted in a 99.9% hit-ratio and VERY high performance on varnish.
	b. And once with a random key that changes the URL (something like 'index.php?rand=193837364'), thus forcing Varnish to hit the backend, and store the multiple objects in memory. 
	c. A combination of the two - in an attempt to maintain a 60-70% hit-ratio.

What we saw is that the kernel simply filled all of the RAM *and* the swap until it crashed. 

2. I'm sorry, but I'm still confused about the mmaped file. If it's limited to 1G, Varnish shouldn't use more than 1G of virtual memory, correct? Or in our setup - 1G of RAM?

Thanks again :)

--
Roi

-----Original Message-----
From: Kristian Lyngstol [mailto:kristian at redpill-linpro.com] 
Sent: Wednesday, October 21, 2009 1:34 PM
To: Roi Avinoam
Cc: varnish-misc at projects.linpro.no
Subject: Re: Varnish virtual memory usage

On Mon, Sep 21, 2009 at 02:55:07PM +0300, Roi Avinoam wrote:
> At Metacafe we're testing the integration with Varnish, and I was 
> tasked with benchmarking our Varnish setup. I intentionally 
> over-flooded the server with requests, in an attempt to see how the 
> system will behave under extensive traffic. Surprisingly, the server 
> ran out of swap and crashed.

That seems mighty strange. What sort of tests did you do?

> In out configuration, "-s file,/var/lib/varnish/varnish_storage.bin,1G".
> Does it mean Varnish shouldn't use more than 1GB of the virtual memory?
> Is there any other way to limit the memory/storage usage?

If you are using -s file and you have 4GB of memory, you are telling Varnish to create a _file_ of 1GB, and it's up to the kernel what it keeps in memory or not. If you actually run out of memory with this setup, you've either hit a bug (need more details first), or you're doing something strange like having the mmaped file (/var/lib/varnish/) in tmpfs with a sizelimit less than 1GB or something along those lines. But I need more details to say anything for certain.

--
Kristian Lyngstøl
Redpill Linpro AS
Tlf: +47 21544179
Mob: +47 99014497