thread pool issues

[Dennis Hendriksen]

Hi there,

We're running Varnish 2.0.6 on a dual quad core server which is doing
about 500 req/s with a 97% hit ratio serving mostly images with. When we
increase the load to about 800 req/s than we encounter two problems that
seem to be related with the thread pool increase.

When we double the varnish load then the "N worker threads limited"
increases rapidly (100k+) while the "N worker threads created" does not
increase (8 pools, min pool size 25, max pool size 1000). Varnish is
unresponsive and client connections hang.

At other times we see the number of worker threads increasing but again
connections 'hang' while Varnish doesn't show any dropped connections
(only overflows).

The backends are fast and can easily handle such kind of load. I've
posted our Varnish config below.
 
Is there anyone out there who knows how to solve our problem? Any help
would be greatly appreciated!

Greetings,
Dennis

varnishadm -T localhost:6082 param.show
accept_fd_holdoff          50 [ms]
acceptor                   default (epoll, poll)
auto_restart               on [bool]
backend_http11             on [bool]
between_bytes_timeout      60.000000 [s]
cache_vbe_conns            off [bool]
cc_command                 "exec cc -fpic -shared -Wl,-x -o %o %s"
cli_buffer                 8192 [bytes]
cli_timeout                5 [seconds]
client_http11              off [bool]
clock_skew                 10 [s]
connect_timeout            0.400000 [s]
default_grace              10
default_ttl                120 [seconds]
diag_bitmap                0x0 [bitmap]
err_ttl                    0 [seconds]
esi_syntax                 0 [bitmap]
fetch_chunksize            128 [kilobytes]
first_byte_timeout         60.000000 [s]
group                      varnish (103)
listen_address             :80
listen_depth               1024 [connections]
log_hashstring             off [bool]
log_local_address          off [bool]
lru_interval               2 [seconds]
max_esi_includes           5 [includes]
max_restarts               4 [restarts]
obj_workspace              8192 [bytes]
overflow_max               100 [%]
ping_interval              3 [seconds]
pipe_timeout               60 [seconds]
prefer_ipv6                off [bool]
purge_dups                 on [bool]
purge_hash                 on [bool]
rush_exponent              3 [requests per request]
send_timeout               600 [seconds]
sess_timeout               5 [seconds]
sess_workspace             16384 [bytes]
session_linger             50 [ms]
session_max                100000 [sessions]
shm_reclen                 255 [bytes]
shm_workspace              8192 [bytes]
srcaddr_hash               1049 [buckets]
srcaddr_ttl                0 [seconds]
thread_pool_add_delay      20 [milliseconds]
thread_pool_add_threshold  2 [requests]
thread_pool_fail_delay     200 [milliseconds]
thread_pool_max            1000 [threads]
thread_pool_min            25 [threads]
thread_pool_purge_delay    1000 [milliseconds]
thread_pool_stack          unlimited [bytes]
thread_pool_timeout        300 [seconds]
thread_pools               8 [pools]
user                       varnish (101)
vcl_trace                  off [bool]