varnishd fails under high load

[Andrew Knapp]

Hey,

 

Sorry if this is the wrong place to be posting this, but I didn't see
any other lists that seemed to fit the question.

 

Basically, we are currently using varnish for some our photo caching,
and it's working perfectly. Here's the relevant portion of varnishstat
from a photo cache:

 

client_conn           4478824        49.86 Client connections accepted

client_req           10146386       112.96 Client requests received

cache_hit             9101426       101.32 Cache hits

cache_hitpass              27         0.00 Cache hits for pass

cache_miss            1044704        11.63 Cache misses

backend_conn          1044954        11.63 Backend connections success

backend_fail                0         0.00 Backend connections failures

backend_reuse          979195        10.90 Backend connections reuses

backend_recycle        985646        10.97 Backend connections recycles

backend_unused             39         0.00 Backend connections unused

 

We start varnishd up with this commandline:

/usr/local/sbin/varnishd -a :80 -f /etc/varnish/default.vcl -T
127.0.0.1:6082 -t 120 -w 1,1000,120 -s
file,/c01/varnish/varnish_storage.bin,40% -u varnish -g varnish -n
/var/varnish -P /var/run/varnish.pid

 

Varnish is working exactly as we expected with photos, and exceptionally
better than squid (which we also use). We have a problem when we use
varnish for serving audio and video files though. Using the same
commandline, varnish eats up all the memory in the system, and then
starts to eat into swap, which in turn makes the system unusable. During
our test, the load rose on the machine to ~950 (!), at which point we
shut it off and went back to using squid, where the load hovers around
~6.

 

The machine running squid is a dell pe1950, 8gb of ram, 1 xeon 2.33 ghz.
The pe1950 has 4 drives in the system formatted as a raid 10, and is
connected to a md1000 (external storage) consisting of 15 73gb 15k scsi
drives formatted in a raid 10+HS. This is formatted at /c01, which, as
you can see above, is where we keep the backend storage for varnish.

 

Unfortunately, I don't have the output from varnish on the audio/video
system. I'm just wondering if anyone has any suggestions on anything I
may be doing wrong. Is 40% (~190GB) too much for a backend storage? The
default.vcl file on the audio/video system is about as simple as you
get, with something added for TTL's. I know varnish is being used on
some pretty big sites out there, so I'm hoping people have some
pointers...

 

Thanks,

 

--

Andrew Knapp | systems engineer | imeem, inc. | 415.762.0135 o |
415.994.5752 m | 415.836.0088 f

 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/varnish-misc/attachments/20070803/a94101fc/attachment-0001.html>