Thread pools

[Mark Moseley]

On Wed, Aug 19, 2009 at 3:54 AM, Poul-Henning Kamp<phk at phk.freebsd.dk> wrote:
> In message <294d5daa0908171755y44f5c132o587f3c8188493a4 at mail.gmail.com>, Mark M
> oseley writes:
>>I've seen various things in the wiki and threads on this list talking
>>about thread pools. In general, the advice is typically conservative,
>>i.e. don't use more than the default 2 thread pools unless you have
>>to. I've also seen the occasional comment suggesting one run as many
>>thread pools as there are cores/cpus.
>
> I think the point here is "don't make 1000 or even 100 pools".
>
> One pool per core should be all you need to practically eliminate
> thread contention, but to truly realize this, we would have to pin
> pools on cores and other nasty and often backfiring "optimizations".
>
> Having a few too many pools probably does not hurt too much, but
> may increase the thread create/kill ratio a bit.
>
>>Also, the wiki mentions that it's mainly appropriate when you run into
>>locks tying things up. Is that mainly a case of high LRU turnover or
>>are there other scenarios where locking is an issue? What are the
>>symptoms of locking becoming an issue with the current configuration
>>and what fields in varnishstat should I be looking at?
>
> POSIX unfortunately does not offer any standard tools for analysing
> lock behaviour, so we have had to make some pretty crude ones
> ourselves.
>
> The main sign of lock issues is that the number of context switches
> increase drastically, you OS can probably give you a view of that
> number.
>
> If you want to go deeper, we have a flag in diag_bitmaps that enables
> shmlogging of lock contentions (or even all lock operations), together
> with varnishtop suitably filtered, that gives a good idea which locks
> we have trouble with.
>
>>but I worry about wasting any CPU% on those 8 core 1950s [...]
>
> Varnish is all about wasting CPU%, normally we barely touch the
> CPUs, man systems running with 80-90% idle CPUs.
>
> Have you played a bit with varnihshist ?   I suspect that may be
> the most sensitive, if crude, indicator of overall performance
> we can offer.

This is all excellent info, lots to chew on. Thanks! I've stared at
varnishhist a number of times and both our hits and misses are showing
up in pretty expected places.