Strategy for large cache sets

[Sascha Ottolski]

Am Dienstag 01 Juli 2008 20:16:15 schrieb Skye Poier Nott:
> I want to deploy Varnish with very large cache sizes (200GB or more)
> for large, long lived file sets.  Is it more efficient to use large
> swap or large mmap in this scenario?
>
> According to the FreeBSD lists, even 20GB of swap requires 200MB of
> kern.maxswzone just to keep track of it, so it doesn't seem like that
> will scale too well.  Is one or the other method better for many
> small files vs less many big files?
>
> Thanks again... when I'm a Varnish expert I'll help the newbs :)
>
> Skye

I'm administrating three varnish instances for static images; cache file 
is setup as 517 GB (pre allocated with "dd"); currently, after 35 days 
uptime, it's filled more than half:

305543155712 bytes allocated
248759787520 bytes free

It's running on Debian etch, 2x dual-core amd opteron, 32 GB RAM, 30 GB 
swap:

Mem:  32969244k total, 32870944k used,    98300k free,   120672k buffers
Swap: 29045480k total,  6096612k used, 22948868k free, 25752752k cached

the varnish process looks like this in "top":

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  P COMMAND         
 3221 username  15   0  522g  27g  21g S    1 87.0 539:33.89 2 varnishd         

hitrate is 98% for recent short term, overall average is 95%.

load of the machines is between 1 and 2 on medium traffic times, and 
goes up to 3 only seldom.

but, the request rate is may be relatively low, compared to what others 
reported on the list. I don't have hard numbers unfortunately, average 
is 80.49 according to varnishstat. At peaks the reate may be in the 
order of may be 400 req/sec for a single instance.

We had issues with 1.1.2 crashing, but since running on trunk (r2640), 
everything runs smooth. response time according to a nagios http 
response check is between 0.5 - 1 seconds, almost never over 1 second, 
even at peak times.

Hope it's useful for someone, let me know if you need more details.


Cheers, Sascha