Varnish burns the CPU and eat the RAM

[Caunter, Stefan]

Hello:

You do not have 

return(lookup);

in recv, not sure why, but that seems odd. Try it with that and see what happens with the test.

We (have to) assume this is a 64bit OS.

-----Original Message-----
From: varnish-misc-bounces at varnish-cache.org [mailto:varnish-misc-bounces at varnish-cache.org] On Behalf Of Romain LE DISEZ
Sent: March-02-11 11:46 AM
To: varnish-misc at varnish-cache.org
Subject: Varnish burns the CPU and eat the RAM

Hello all,

I'm pretty new to Varnish. I'm deploying it because one of our customer is going to have a special event and the website is pretty slow (I'm working for an Internet hosting company). We are expecting more than 1000 requests per seconds.

From what I read here and there, this should not be a problem for Varnish.

My problem is that when Varnish is using cache ("deliver", as opposed to "pass"), the CPU consumption increases drasticaly, also the RAM.

The server is a Xeon QuadCore 2.5Ghz, 8GB of RAM.


With a simple test like this (robots.txt = 300 bytes) :
    ab -r -n 1000000 -c 1000 http://www.customer1.com/robots.txt

CPU consumption is fluctuating between 120% and 160%.

Second point is that Varnish consumes all the memory. Trying to limit that, I made a tmpfs mountpoint of 3G :
    mount -t tmpfs -o size=3g tmpfs /mnt/varnish/

But varnish continues to consume all the memory

My configuration is attached to this mail. Varnish is launched like this :
/usr/sbin/varnishd -P /var/run/varnish.pid
    -a :80
    -f /etc/varnish/default.vcl
    -T 127.0.0.1:6082
    -t 120
    -w 120,120,120
    -u varnish -g varnish
    -S /etc/varnish/secret
    -s file,/mnt/varnish/varnish_storage.bin,100%
    -p thread_pools 4

I also tried to launch it with parameter "-h classic"

It is installed on a Centos 5 up to date, with lastest RPMs provided by the varnish repository.

If I put a return (pass) in vcl_fetch, everything is fine (except the backend server, of course). It makes me think, with my little knowledges of Varnish, that the problem is in the delivering from cache.

Output of "varnishstat -1", when running ab, is attached.

Thanks for your help.


--
Romain LE DISEZ