High LRU nuked with plenty of cache space available

[Dan Crosta]

We're running varnish (3.0.4) like:

/usr/sbin/varnishd -P /var/run/varnish.pid -a :80 -f
/etc/varnish/default.vcl -T 127.0.0.1:8080 -t 2628000 -w 50,1000,120
-u varnish -g varnish -S /etc/varnish/secret -s
file,/srv/varnish/varnish_storage.bin,12G -p thread_pools=4 -p
http_gzip_support=off -p saintmode_threshold=0 -p
http_range_support=off

varnishstat shows the following for SMF:

SMF.s0.c_req            45213612       317.34 Allocator requests
SMF.s0.c_fail           14781854       103.75 Allocator failures
SMF.s0.c_bytes      2056164147200  14431753.97 Bytes allocated
SMF.s0.c_freed      2052654858240  14407123.06 Bytes freed
SMF.s0.g_alloc            856760          .   Allocations outstanding
SMF.s0.g_bytes        3509288960          .   Bytes outstanding
SMF.s0.g_space        9375612928          .   Bytes available
SMF.s0.g_smf              950080          .   N struct smf
SMF.s0.g_smf_frag          93319          .   N small free smf
SMF.s0.g_smf_large             1          .   N large free smf

which I interpret to mean that ~9G of the 12G are available ("bytes
available") for use by the cache.

However, when our applications service the API through varnish at a
rate of about 150 QPS, we see that n_lru_nuked increases at about the
same rate, the result of which is that the n_object counter stays more
or less constant. The cache has only been running for a few days, so I
don't believe TTL is to blame here, since we use -t with about 30 days
(assuming the value there is in seconds, which the docs seem to
imply).

I have tried setting a granularity on the storage, which did not seem
to have any impact. I'm looking for other suggestions or things to
try, as intuition and the stats seem to suggest we should be able to
store a lot more objects before things start to get nuked.

Thanks,
- Dan