varnish is using all file descriptors

[Per Buer]

Hi Vince.

There are to limits on file descriptors on a Linux system. There is
the global one, which was the one I thought you have been adjusting (
/proc/sys/kernel/fs/something) and the per process tree one, rlimits
(kernel part), also known as ulimits (user space part). A Varnish
server will need quite a few FDs of the first kind, not so many of the
second.

So, you're right, you have a problem. Check to see whether the same
problem is present at the backend. You might have a slow backend that
is piling up connections. If so, you could work around the problem by
setting some very agressive timeouts. You'll serve some errors, but at
least the site won't go down.


Per.



On Mon, Jun 20, 2011 at 10:08 PM, Vince <cooltechemail at gmail.com> wrote:
> We are using CentOS 5.6 64-bit
> I'll try set it to a higher number. Do we need to turn any kernel parameters
> if we set it to a really high number, say 1M, on CentOS?
> However, a higher number doesn't mean we have found the problem. Our varnish
> server is serving about 300-400 requests per second. When it runs ok we
> checked /proc/PID/fd and it uses less than 100 file descriptors. But when it
> goes wrong all 65536 file descriptors have been used. Is this normal?
> Thank you!
>
> On Tue, Jun 21, 2011 at 4:02 AM, Per Buer <perbu at varnish-software.com>
> wrote:
>>
>> On Mon, Jun 20, 2011 at 9:31 PM, Vince <cooltechemail at gmail.com> wrote
>> > Hi,
>> > We had a problem with our varnish server recently that sometimes the
>> > server
>> > will use all file descriptors available to it, which is set to 65536,
>> > and
>> > start refusing connections. I am wondering how to find out what's the
>> > real
>> > problem behind it.
>>
>> Why the insanely low number? Out of the box my laptop has over half a
>> million. With 2minutes timeout for reuse on a tcp socket you would run
>> of sockets in a jiffie - oh, wait, that's what happening. :-)
>>
>> I would set up waaaay up. FD's are really cheap. You could start to
>> tune your TCP stack and/or reduce the number of Varnish threads but
>> unless you are on a embedded system I wouldn't do it.
>>
>> --
>> Per Buer, CEO
>> Phone: +47 21 98 92 61 / Mobile: +47 958 39 117 / Skype: per.buer
>> Varnish makes websites fly!
>> Whitepapers | Video | Twitter
>
>



-- 
Per Buer, CEO
Phone: +47 21 98 92 61 / Mobile: +47 958 39 117 / Skype: per.buer
Varnish makes websites fly!
Whitepapers | Video | Twitter