Connections dropped under load

[Stig Bakken]

This seems similar to what I've been seeing, described in an earlier thread
from before christmas.  In my case it was not during benchmarking, but when
serving production load of around 300 req/s per server.  Modern tcpip stacks
on modern hardware should handle this without blinking.

Did you have the chance to capture the problem with varnishlog so you can
replay/analyze it?

 - Stig

On Wed, Jan 5, 2011 at 11:18 PM, George Georgovassilis <
g.georgovassilis at gmail.com> wrote:

> I removed the varnish instance so that the load generator is directly
> hitting Tomcat. Naturally, the request rate drops to 70 requests/sec with a
> CPU load of 100%... however connections don't drop anymore, no timeouts
> occur and the application remains pretty responsive. To recap, these are the
> possible scenarios:
>
> 1. The networking layer is overtaxed with the original 300 reqs/sec. I
> don't believe that, because the load generator doesn't record any dropped
> connections while a simple browser can't connect.
>
> 2. Tomcat is overtaxed. That also seems not plausible, since it is not
> servicing any requests under the load test - all is done by varnish. Even
> if, as I said when removing varnish from in between, it serves the requests
> just fine.
>
> 3. Varnish is overtaxed. Somehow that also doesn't make sense, since it is
> servicing the load generator just fine... but will refuse to serve browser
> requests.
>
> 4. Varnish, when under load, is picky about what connections to serve.
>
> I'm stuck :-)
>
>
> On 05.01.2011 17:59, Bob Camp wrote:
>
>> Hi
>>
>> Running simple load tests both on Apache directly, and on Varnish - both
>> seem to experience "long delays" on a small percentage of the requests.
>> The
>> problem does not appear to happen with low loads. It does come up as CPU
>> usage becomes an issue. It also is hard to make happen with a single
>> stream
>> of requests. It seems to come up much quicker with many requests done in
>> parallel.
>>
>> I've always *assumed* that the poor little TCP/IP hamster simply ran out
>> of
>> breath and started dropping connections.
>>
>> Bob
>>
>> -----Original Message-----
>> From: varnish-misc-bounces at varnish-cache.org
>> [mailto:varnish-misc-bounces at varnish-cache.org] On Behalf Of George
>> Georgovassilis
>> Sent: Wednesday, January 05, 2011 11:18 AM
>> To: varnish-misc at projects.linpro.no
>> Subject: Re: Connections dropped under load
>>
>> Hello Cosimo,
>>
>> Thank you for the quick reply. After your hint I had the tests run again
>> but couldn't detect that pattern. What susprised me though after looking
>> through the logs is that almost all requests by the load generator
>> complete in a timely manner (<  1 sec), but all requests generated by a
>> real browser (IE, FF, Opera) will be served much later or even run into
>> a timeout.
>>
>> On 05.01.2011 16:30, Cosimo Streppone wrote:
>>
>>> On Wed, 05 Jan 2011 16:20:31 +0100, George Georgovassilis
>>> <g.georgovassilis at gmail.com>  wrote:
>>>
>>>  I'm having trouble with dropped connections under a loadtest.
>>>>
>>>> The problem: As a measure for response, I am requesting an image from
>>>> the webapp running in Tomcat while the loadtest is underway. However
>>>> that either times out or is delivered after several seconds. Varnishlog
>>>> will often either not show the request (RxURL) at all, or show it
>>>> several seconds after the browser dispatched it.
>>>>
>>> Hi George,
>>>
>>> if you measure the time you mention as "several seconds"
>>> and it's either 3 or 9 seconds, I think what you're seeing
>>> is a client-side TCP retransmit timeout.
>>>
>>> I experienced that, both under load testing,
>>> and in real production setups.
>>>
>>>
>> _______________________________________________
>> varnish-misc mailing list
>> varnish-misc at varnish-cache.org
>> http://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
>>
>>
>
> _______________________________________________
> varnish-misc mailing list
> varnish-misc at varnish-cache.org
> http://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
>



-- 
Stig Bakken
CTO, Zedge.net - free your phone!
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/varnish-misc/attachments/20110105/ff7f83c2/attachment-0003.html>