abnormally high load?

[Mark Moseley]

On Wed, Aug 12, 2009 at 12:25 PM, Ken Brownfield<kb+varnish at slide.com> wrote:
> My first guess is that you're seeing varnish spawn a lot of threads
> because your back-end isn't keeping up with the miss rate.  My second
> guess is that these misses are large files that are taking a long time
> for clients to download, therefore piling up active client connections
> (and thus worker threads).
>
> I'm guessing your load is going high because you're swapping?  In top,
> are your CPUs saturated, or fairly idle?
>
> If you're seeing CPU saturation, this is possibly an internal Varnish
> issue.  Your VCL seems sane, but we haven't seen the inline C.
>
> It's a fact of life that you may periodically need more back-end or
> worker threads than you would normally see.  If you're on Linux (at
> least), each of those threads will use 8MB of RAM (the default stack
> size) which adds up quickly.  You can reduce the thread stack size to
> dramatically decrease how much memory Varnish uses as it scales threads.
>
> We run a patch here that adds a startup parameter to change the stack
> size of backend and worker pthreads, but you could emulate this by
> reducing stack size before running Varnish:
>
>        ulimit -s 256
>          or
>        limit stacksize 256
>
> We run pretty heavy traffic (including inline C) with 256KB stack with
> no problem.  This adjustment should decrease memory usage as thread
> counts increase, and if you're swapping it might help alleviate the
> spikes.  But if the spikes are due to a slow backend, that's probably
> where I'd look first.
>
> Hope it helps,
> --
> Ken


Not looking to hijack the thread, but that got my attention. Is there
any rule of thumb that would determine how small of a stack size a
varnish could away with and still perform ok? I dropped our stack size
to 1 meg, which seemed drastic at the time. But if you're doing 256k,
that's even better. We're using varnish in a web hosting environment
on 32-bit Debian Etch boxes (without hope of going 64-bit anytime
soon), so I can cram about 800 threads total without hitting the 3gb
limit. Traffic patterns are pretty much random, i.e. size of requests
are all over the board, though the average is about 40k. Hit rate is
~25% (which sounds awful but for web hosting, we're overjoyed). Any
red flags for 256kb threads there?