Debugging / nuked objects spike

[Cloude Porteus]

We've been running varnish in production for about a week and I've
been noticing that things aren't quite right, but it's been hard to
figure out what. Most of the time Varnish is running with a 98% hit
ratio and all is fine. We have been running for a few days with about
250k documents in the cache.

I just noticed that the number of documents in the cache dropped from
~140k -> ~30k and the LRU Nuked Objects increased by 100k. I assume
we're hitting our storage limit, which is currently set to 10gb. We
had it set at 50gb before, but we were still having similar problems.
I noticed last night there was a couple of hours where it looked like
the hit ratio was close to zero, but then it went back to normal.

Any ideas what would cause Varnish to nuke ~100k objects all at once?
I've gone over all the performance tuning info and we've tried to
implement most of the suggestions. I'm just not sure which direction
to start tuning further.

Thanks for any suggestions. Here's our current default.vcl:
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
# Configuration file for varnish
NFILES=131072
MEMLOCK=82000
VARNISH_VCL_CONF=/etc/varnish/instructables.vcl
VARNISH_LISTEN_ADDRESS=
VARNISH_LISTEN_PORT=80
VARNISH_ADMIN_LISTEN_ADDRESS=127.0.0.1
VARNISH_ADMIN_LISTEN_PORT=82
VARNISH_MIN_THREADS=400
VARNISH_MAX_THREADS=1000
VARNISH_THREAD_TIMEOUT=60
VARNISH_STORAGE_FILE=/var/lib/varnish/varnish_storage.bin
VARNISH_STORAGE_SIZE=10G
VARNISH_STORAGE="file,${VARNISH_STORAGE_FILE},${VARNISH_STORAGE_SIZE}"
VARNISH_TTL=1800

DAEMON_OPTS="-a ${VARNISH_LISTEN_ADDRESS}:${VARNISH_LISTEN_PORT} \
             -f ${VARNISH_VCL_CONF} \
             -T ${VARNISH_ADMIN_LISTEN_ADDRESS}:${VARNISH_ADMIN_LISTEN_PORT} \
             -t ${VARNISH_TTL} \
             -w
${VARNISH_MIN_THREADS},${VARNISH_MAX_THREADS},${VARNISH_THREAD_TIMEOUT}
\
             -u varnish -g varnish \
             -s ${VARNISH_STORAGE} \
             -p obj_workspace=4096 \
             -p sess_workspace=262144 \
             -p lru_interval=3600
             -p listen_depth=8192 \
             -p log_hashstring=off \
             -p sess_timeout=10 \
             -p shm_workspace=32768 \
             -p ping_interval=1 \
             -p thread_pools=4 \
             -p thread_pool_min=100 \
             -p srcaddr_ttl=0 \
             -p esi_syntax=1 "

thanks,
cloude
-- 
VP of Product Development
Instructables.com

http://www.instructables.com/member/lebowski