Varnish caching issues, cache size, expires, cache resets, whats going on?

[Demitrious Kelly]

Hello,

We have a varnish (varnish-1.1.2) server handling about 2,500 requests 
per second.  The nature of these requests are extraordinarily random, 
and the disks cannot keep up with the demand, we therefor must keep the 
cache in RAM.  Under highest traffic conditions this generates about 3GB 
per hour of cached data.  The expires on the data being generated are 
set to a random number of seconds between 1800 and 3600.  So 
theoretically the cache size should never reach more than 4gb or so, 
well within the servers RAM capacity of 8Gb.  What we are seeing however 
is that the cache will continue to grow to (and beyond if we allow it,) 
the 8gb mark.  At which point we have to invalidate the cache by 
restarting the processes (naturally this causes the undesired effect of 
pegging the back-end servers with very high loads until enough data is 
back in the cache).  We have tried limiting the cache size to 7gb, 
however once the 7gb mark is reached the varnish cache restarts... with 
the undesired effect of pegging the back-end servers with very high 
loads until enough data is back in the cache.

One definite oddity in our particular usage patterns is that 30% or more 
of our cached data is comprised of headers, and not content.  Currently, 
for example, varnishstat tells me:
    3424095028    743539.36    851553.10 Total header bytes
    7830249685   1718569.76   1947338.89 Total body bytes

Does anybody have any insight into this behavior?  We would like to not 
have to resort to multiple varnish servers with mickey mouse hacks like 
staggering restarts to reduce the effect on the back end servers.

System details below:
---------------------------------------

root at foo.com:~# uname -a
    Linux foo.com 2.6.18-6-amd64 #1 SMP Sun Feb 10 17:50:19 UTC 2008 
x86_64 GNU/Linux

root at foo.com:~# free -m
                 total       used       free     shared    buffers     
cached
    Mem:          7991       4708       3282          0         91       
2317
    -/+ buffers/cache:       2300       5691
    Swap:            0          0          0

root at foo.com:~# cat /proc/cpuinfo  | grep -E 'mhz|model name' -i
    model name      : Dual-Core AMD Opteron(tm) Processor 2218
    cpu MHz         : 2593.784
    model name      : Dual-Core AMD Opteron(tm) Processor 2218
    cpu MHz         : 2593.784
    model name      : Dual-Core AMD Opteron(tm) Processor 2218
    cpu MHz         : 2593.784
    model name      : Dual-Core AMD Opteron(tm) Processor 2218
    cpu MHz         : 2593.784

Start Command:
    /usr/local/sbin/varnishd -a 0.0.0.0:80 -f /etc/varnish/varnish.vcl 
-P /var/run/varnishd.pid -T 192.x.x.x:6969 -t 600 -w

/etc/varnish/varnish.vcl
    backend default {
      set backend.host = "foo.lan";
      set backend.port = "http";
    }

    sub vcl_recv {
      if ( req.url ~ "^/(part1|part2)" && req.http.cookie )  {
        lookup;
      }
    }

    sub vcl_fetch {
      if (obj.ttl < 600s) {
        set obj.ttl = 600s;
      }
      if (obj.status == 404) {
        set obj.ttl = 10s;
      }
      if (obj.status == 500 || obj.status == 503) {
        pass;
      }
      insert;
    }