Under Load: Server Unavailable/Connection Dropped/Delayed Reponse

[Tejaswi Nadahalli]

On Fri, Mar 4, 2011 at 9:43 AM, Caunter, Stefan <scaunter at topscms.com>wrote:

>
> What does something like firebug show when you request during the load
> test? The delay may be anything from DNS to the ec2 network.
>

The DNS requests are getting resolved super quick. I am unable to see any
other network issues with EC2. I have a similar machine in the same data
center running nginx which is doing similar loads, but with no caching
requirement, and it's running fine.

In my first post, I forgot to attach my VCL, which is a bit too minimal. Am
I missing something obvious?

------
backend default0 {
    .host = "10.202.30.39";
    .port = "8000";
}

sub vcl_recv {
    unset req.http.Cookie;
    set req.grace = 3600s;
    set req.url = regsub(req.url, "&refurl=.*&t=.*&c=.*&r=.*", "");
}

sub vcl_deliver {
  if (obj.hits > 0) {
    set resp.http.X-Cache = "HIT";
  } else {
    set resp.http.X-Cache = "MISS";
  }
}
-------------------------

Could there be some kind of TCP packet pileup that I am missing?

-T


>
>
> Stefan Caunter
>
> Operations
>
> Torstar Digital
>
> m: (416) 561-4871
>
>
>
>
>
> *From:* varnish-misc-bounces at varnish-cache.org [mailto:
> varnish-misc-bounces at varnish-cache.org] *On Behalf Of *Tejaswi Nadahalli
> *Sent:* March-04-11 1:09 AM
> *To:* varnish-misc at varnish-cache.org
> *Subject:* Under Load: Server Unavailable/Connection Dropped/Delayed
> Reponse
>
>
>
> Hi Everyone,
>
> I am seeing a situation similar to :
>
>
> http://www.varnish-cache.org/lists/pipermail/varnish-misc/2011-January/005351.html(Connections Dropped Under Load)
>
> http://www.varnish-cache.org/lists/pipermail/varnish-misc/2010-December/005258.html(Hanging Connections)
>
> I have httperf loading a varnish cache with never-expire content. While the
> load is on, other browser/wget requests to the varnish server get delayed to
> 10+ seconds. Any ideas what could be happening? ssh doesn't seem to be
> impacted. So, is it some kind of thread problem?
>
> In production, I see a similar situation with around 1000 req/second load.
>
> I am running varnishd with the following command line options (as per
> http://kristianlyng.wordpress.com/2009/10/19/high-end-varnish-tuning/):
>
> sudo varnishd -f /etc/varnish/default.vcl -s malloc,5G -T 127.0.0.1:2000-a
> 0.0.0.0:80 -p thread_pools=8 -p thread_pool_min=100 -p
> thread_pool_max=5000 -p thread_pool_add_delay=2 -p cli_timeout=25 -p
> session_linger=100 -p lru_interval=20 -t 31536000
>
> I am on Ubuntu Lucid 64 bit Amazon EC2 C1.XLarge with 8 processing units.
>
> My network sysctl parameters are tuned according to:
> http://varnish-cache.org/trac/wiki/Performance
> fs.file-max = 360000
> net.ipv4.ip_local_port_range = 1024 65536
> net.core.rmem_max = 16777216
> net.core.wmem_max = 16777216
> net.ipv4.tcp_rmem = 4096 87380 16777216
> net.ipv4.tcp_wmem = 4096 65536 16777216
> net.ipv4.tcp_fin_timeout = 3
> net.core.netdev_max_backlog = 30000
> net.ipv4.tcp_no_metrics_save = 1
> net.core.somaxconn = 262144
> net.ipv4.tcp_syncookies = 0
> net.ipv4.tcp_max_orphans = 262144
> net.ipv4.tcp_max_syn_backlog = 262144
> net.ipv4.tcp_synack_retries = 2
> net.ipv4.tcp_syn_retries = 2
>
>
> Any help would be greatly appreciated
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/varnish-misc/attachments/20110304/bd710d22/attachment-0003.html>