Varnish restarts when all memory is allocated

[frank]

Hi Guys,

On Tue, 2009-05-26 at 23:29 +0200, Marco Walraven wrote:

> We are testing a Varnish Cache in our production environment with a 500Gb storage file and
> 32Gb of RAM. Varnish performance is excellent when all of the 32Gb is not allocated yet.
> The rates I am seeing here are around 40-60Mbit/s, with roughly 2.2M objects in cache and
> hitting a ratio of ~0.65, even then Varnish can handle it easily. However it is still
> warming up since we have a lot of objects that need to be cached.
> 
> The problem I am facing is that as soon as RAM is exhausted Varnish restarts itself.
> Since this looked like an IO problem, we dropped ext2 in favour of xfs with much
> better results on writing to disk. However varnishd still stops working after it get
> to the 32G RAM limit. Note that I don't see any IO until just before it hits the 97% of
> RAM usage.      
> 
> So we thought to combine the file storage type with malloc and limit the amount of
> memory Varnish is allowed to allocate, first to 5G and see how that would work out. 
> It turned out that it did not get limited and it seems from reading some posts     
> this is not needed..
> 
> I have seen some posts on running large caches with the same kind but not a real
> approach to a solution. What is the best way to get around this issue ?

I am facing exactly the same problem. Each time Varnish hits the RAM
limit, it starts to drop elements from the cache. During this time the
system is nearly unusable due to heavy disk io. After that Varnish keeps
running but uses just a small part of the available memory.

DAEMON_OPTS="-a :3129 \
             -T localhost:6082 \
             -f /etc/varnish/default.vcl \
             -p thread_pool_max=1500 \
             -p lru_interval=3600 \
             -h classic,500009 \
             -p thread_pools=4 \
             -p listen_depth=4096 \
             -s file,/opt/varnish/cache.bin1,26G"

I tried malloc without any results.
             -s malloc,5G"

If i run Varnish with the default setting it does not use the whole
memory (8G out of 32G)

A second instance which serves other documents runs fine with 1GB cache
file and the default settings. Same hardware with debian 4.0 and
2.6.18-6-amd64

Am I missing something? Is there a rule of thumb for calculating the
cache file / size? How does Varnish flush objects from the cache?

Thanks in advance

Frank