Slow connections

[Michael S. Fischer]

That kind of VM overcommit (400GB on an 8GB box) is hazardous for performance anyway.  I strongly advise configuring Varnish cache sizes at slightly under the actual RAM size of the box.  If your working set size is larger, you need more boxes or more RAM anyway, as paging I/O will significantly impede your throughput.

Another thing that would be useful to know are the TCP stats from the haproxy, e.g., netstat -s -t (Linux).  Unfortunately you can't determine retransmits on a per-destination basis using netstat alone, but there might be open source tools out there that can help. 

Finally, sometimes I've seen these issues where the network interface or the switch were having speed/duplex autonegotiation issues or bad network cables; manually configuring the switch (e.g. to 1Gb full duplex) sometimes solved the issue.

--Michael

On Dec 28, 2009, at 12:55 PM, Joe Williams wrote:

> Thanks Cosimo, I'll have to give that a shot. I use 400GB because I have 
> a dedicated EC2 machine for varnish and figured I would use as much 
> space as I have for it, looks like I'm actually only using around 8GB.
> 
> -Joe
> 
> 
> On 12/28/09 12:45 PM, Cosimo Streppone wrote:
>> On 28 Dec 2009 19:59:02, Joe Williams <joe at joetify.com> wrote:
>> 
>>> Any other suggestions? Another analysis of the logs shows that varnish
>>> vs other backends (CouchDB) I see an order of magnitude higher
>>> percentage of 3 second connection times with varnish.
>> 
>>> Here's my varnish command options
>>> http://friendpaste.com/1Zvj1XkX75eKqdzrJQZcfK  and my sysctl changes
>>> http://friendpaste.com/3lt3uhIm1NTNhrqGxvQEX6 the sysctl on haproxy is
>>> identical.
>> 
>> Hi Joe,
>> 
>> I didn't follow all the discussion, but a quick look made me
>> think about my case.
>> 
>> I practically did the same steps, with the same results
>> as yours. Kernel tuning, somaxconn, listen_depth, etc...
>> 
>> In my case I was experiencing dropped connections, no
>> or very delayed syn-ack packets from varnish server, random varnish
>> restarts (crashes?), and sudden system load spikes,
>> even as high as 200/300.
>> 
>> The solution to this problem was really simple.
>> Switch from the "file" allocation to "malloc".
>> In the config file I had:
>> 
>>  -s file,/var/lib/varnish/varnish.cache,20G
>> 
>> and I changed it to:
>> 
>>  -s malloc,20G
>> 
>> I see you have 400G, so this solution won't probably work
>> for you?, but my suggestion would be to at least try it
>> with as much RAM as you have, to see if the issue disappears.
>> 
>> Then you can setup some swap partition maybe.
>> Check this mail,
>> 
>> http://www.mail-archive.com/varnish-misc@projects.linpro.no/msg01571.html
>> 
> 
> -- 
> Name: Joseph A. Williams
> Email: joe at joetify.com
> Blog: http://www.joeandmotorboat.com/
> 
> _______________________________________________
> varnish-misc mailing list
> varnish-misc at projects.linpro.no
> http://projects.linpro.no/mailman/listinfo/varnish-misc