Under Load: Server Unavailable/Connection Dropped/Delayed Reponse

[Caunter, Stefan]

Check your ec2 network settings.

OS and varnish settings look okay, your varnishstat shows that varnish
is coasting along fine.

It's not threads. You have 800 available, according to the varnishstat;
it's running with 800 threads, handling 12,000+ connections, and there
is no thread creation failure. Therefore it does not need to add
threads.

What does something like firebug show when you request during the load
test? The delay may be anything from DNS to the ec2 network.

 

Stefan Caunter 

Operations

Torstar Digital

m: (416) 561-4871

 

 

From: varnish-misc-bounces at varnish-cache.org
[mailto:varnish-misc-bounces at varnish-cache.org] On Behalf Of Tejaswi
Nadahalli
Sent: March-04-11 1:09 AM
To: varnish-misc at varnish-cache.org
Subject: Under Load: Server Unavailable/Connection Dropped/Delayed
Reponse

 

Hi Everyone,

I am seeing a situation similar to : 

http://www.varnish-cache.org/lists/pipermail/varnish-misc/2011-January/0
05351.html (Connections Dropped Under Load)
http://www.varnish-cache.org/lists/pipermail/varnish-misc/2010-December/
005258.html (Hanging Connections)

I have httperf loading a varnish cache with never-expire content. While
the load is on, other browser/wget requests to the varnish server get
delayed to 10+ seconds. Any ideas what could be happening? ssh doesn't
seem to be impacted. So, is it some kind of thread problem?

In production, I see a similar situation with around 1000 req/second
load. 

I am running varnishd with the following command line options (as per
http://kristianlyng.wordpress.com/2009/10/19/high-end-varnish-tuning/):

sudo varnishd -f /etc/varnish/default.vcl -s malloc,5G -T 127.0.0.1:2000
-a 0.0.0.0:80 -p thread_pools=8 -p thread_pool_min=100 -p
thread_pool_max=5000 -p thread_pool_add_delay=2 -p cli_timeout=25 -p
session_linger=100 -p lru_interval=20 -t 31536000

I am on Ubuntu Lucid 64 bit Amazon EC2 C1.XLarge with 8 processing
units.

My network sysctl parameters are tuned according to:
http://varnish-cache.org/trac/wiki/Performance
fs.file-max = 360000
net.ipv4.ip_local_port_range = 1024 65536
net.core.rmem_max = 16777216
net.core.wmem_max = 16777216
net.ipv4.tcp_rmem = 4096 87380 16777216
net.ipv4.tcp_wmem = 4096 65536 16777216
net.ipv4.tcp_fin_timeout = 3
net.core.netdev_max_backlog = 30000
net.ipv4.tcp_no_metrics_save = 1
net.core.somaxconn = 262144
net.ipv4.tcp_syncookies = 0
net.ipv4.tcp_max_orphans = 262144
net.ipv4.tcp_max_syn_backlog = 262144
net.ipv4.tcp_synack_retries = 2
net.ipv4.tcp_syn_retries = 2


Any help would be greatly appreciated

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/varnish-misc/attachments/20110304/0f41914c/attachment-0003.html>