varnish stopped resonding

[Andreas Plesner Jacobsen]

On Wed, Sep 21, 2011 at 10:15:51AM -0700, Matt Schurenko wrote:
> 
> I'm running two varnish servers in production (ver  2.1.5). Both are using
> the same hardware and have the same amount of RAM (48GB). Last night one of
> the varnish servers stopped responding on port 80. Since we are using HAproxy
> in front of both varnish servers for load balancing this did not have much
> effect on our end users. The symptoms of the problem were either a client (
> HAproxy, telnet) could not establish a layer 4 connection to varnish or, if a
> client could establish a connection and issued an HTTP GET, varnish returned
> nothing, no HTTP headers, nothing.

Clients are stuck in the listen queue because you've run out of workers.
Are you sure that the workers are not all stuck trying to get content from the
backend?
You may be experiencing workers piling up on uncacheable urls or urls, that are
being refreshed (where the queue will wait for the first request to complete
unless you're using grace)

You may also want to consider using malloc instead of file storage.

-- 
Andreas