Reducing overflowed and dropped work requests

[Simon Males]

On Wed, Aug 3, 2011 at 5:05 PM, Tollef Fog Heen
<tfheen at varnish-software.com> wrote:
> ]] Simon Males
>
> | Since this post, I increased the thread_pools to 8, though today I've
> | had another spurt of dropped work requests. It's definitely related to
> | a sudden rise of traffic, as the dropped requests happen moment after
> | sending an E-Mail to our users.
> |
> | The servers have 16 cores, so I've increased the thread_pools to 16
> | now. With 10 minimum threads per pool, I now have 160 threads waiting
> | to serve.
>
> If you're on 2.1 you might want to set thread_pool_add_delay to 2 (ms)
> or similar.  It's set a bit too high there, which means it takes a
> little while for Varnish to respond to rapid spikes.

Just wanted to say thanks to the contributors to this thread Václav
Bílek and Tollef Fog Heen.

I actually held back on the advice and wanted to see how far 160
minimum threads would get me (16 pools x 10 minimum). The next spike
produced 320k limited worker threads (n_wrk_max) per server.

I set the following parameters on the fly hoping to manage the spike.
But the real test will be the next spike.
thread_pool_add_delay 2
thread_pool_max 250
thread_pool_min 100

Though now my committed memory usage (according to Munin) is 16G. The
server has a total of 4G and 2G malloc cache. Is there an equation I
can use to determine how much memory I need with my current
parameters?

Additionally can I deduce from varnishstats how many threads are being
used? e.g. How many of the minimum 1600 are being used.

-- 
Simon Males