Varnish stuck on most served content

[Diego Roccia]

Hi Guys,
   This is my first message in this list, I began working for a new 
company some months ago and I found this infrastructure:

+---------+  +---------+  +---------+  +---------+
| VARNISH |  | VARNISH |  | VARNISH |  | VARNISH |
+---------+  +---------+  +---------+  +---------+
      |            |            |            |
      +------------+------------+------------+
                   |            |
            +------+-+       +--+-----+
            | APACHE |       | APACHE |
            +--------+       +--------+

Varnish servers are HP DL360 G6 with 66Gb RAM and 4 Quad-Core Xeon CPUs, 
running varnish 2.1.6 (Updated from 2.0.5 1 month ago). They're serving 
content for up to 450Mbit/s during peaks.

It's happening often that they freeze serving contents. and I noticed a 
common pattern: the content that get stuck is always one of the most 
served, like a css or js file, or some component of the page layout, and 
it never happens to an image part of the content.

It's really weird, because css should be always cached.

I'm running Centos 5.5 64bit and here's my varnish startup parameters:

DAEMON_OPTS=" -a ${VARNISH_LISTEN_ADDRESS}:${VARNISH_LISTEN_PORT} \
		-f ${VARNISH_VCL_CONF} \
		-T 0.0.0.0:6082 \
		-t 604800 \
		-u varnish -g varnish \
		-s malloc,54G \
		-p thread_pool_add_delay=2 \
                 -p thread_pools=16 \
                 -p thread_pool_min=50 \
                 -p thread_pool_max=4000 \
		-p listen_depth=4096 \
		-p lru_interval=600 \
		-hclassic,500009 \
		-p log_hashstring=off \
		-p shm_workspace=16384 \
		-p ping_interval=2 \
		-p default_grace=3600 \
		-p pipe_timeout=10 \
		-p sess_timeout=6 \
		-p send_timeout=10"

In attach there is my vcl and the varnishstat -1 output after a 24h run 
of 1 of the servers. Do you notice something bad?

In the meanwhile I'm running through the documentation, but it's for us 
an high priority issue as we're talking about the production environment 
and there's not time now to wait for me to completely understand how 
does varnish work and find out a solution.

Hope someone can help me
Thanks in advance
Diego





-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: stats.txt
URL: <http://www.varnish-cache.org/lists/pipermail/varnish-misc/attachments/20110330/624ee8d5/attachment.txt>
-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: default.vcl
URL: <http://www.varnish-cache.org/lists/pipermail/varnish-misc/attachments/20110330/624ee8d5/attachment.ksh>