Connections dropped under load

Wed Jan 5 23:56:41 CET 2011

This thread:
http://www.varnish-cache.org/lists/pipermail/varnish-misc/2010-December/005258.html

 - Stig

On Wed, Jan 5, 2011 at 11:44 PM, George Georgovassilis <
g.georgovassilis at gmail.com> wrote:

>  Hello Stig,
>
> Thanks for the insight. I'm still on the logs, though not sure where to
> start - it's not like that there are any errors in it so I'm not really sure
> what to look for. Do you have a pointer to that discussion you are referring
> to?
>
>
> On 05.01.2011 23:41, Stig Bakken wrote:
>
> This seems similar to what I've been seeing, described in an earlier thread
> from before christmas.  In my case it was not during benchmarking, but when
> serving production load of around 300 req/s per server.  Modern tcpip stacks
> on modern hardware should handle this without blinking.
>
>  Did you have the chance to capture the problem with varnishlog so you can
> replay/analyze it?
>
>   - Stig
>
> On Wed, Jan 5, 2011 at 11:18 PM, George Georgovassilis <
> g.georgovassilis at gmail.com> wrote:
>
>> I removed the varnish instance so that the load generator is directly
>> hitting Tomcat. Naturally, the request rate drops to 70 requests/sec with a
>> CPU load of 100%... however connections don't drop anymore, no timeouts
>> occur and the application remains pretty responsive. To recap, these are the
>> possible scenarios:
>>
>> 1. The networking layer is overtaxed with the original 300 reqs/sec. I
>> don't believe that, because the load generator doesn't record any dropped
>> connections while a simple browser can't connect.
>>
>> 2. Tomcat is overtaxed. That also seems not plausible, since it is not
>> servicing any requests under the load test - all is done by varnish. Even
>> if, as I said when removing varnish from in between, it serves the requests
>> just fine.
>>
>> 3. Varnish is overtaxed. Somehow that also doesn't make sense, since it is
>> servicing the load generator just fine... but will refuse to serve browser
>> requests.
>>
>> 4. Varnish, when under load, is picky about what connections to serve.
>>
>> I'm stuck :-)
>>
>>
>> On 05.01.2011 17:59, Bob Camp wrote:
>>
>>> Hi
>>>
>>> Running simple load tests both on Apache directly, and on Varnish - both
>>> seem to experience "long delays" on a small percentage of the requests.
>>> The
>>> problem does not appear to happen with low loads. It does come up as CPU
>>> usage becomes an issue. It also is hard to make happen with a single
>>> stream
>>> of requests. It seems to come up much quicker with many requests done in
>>> parallel.
>>>
>>> I've always *assumed* that the poor little TCP/IP hamster simply ran out
>>> of
>>> breath and started dropping connections.
>>>
>>> Bob
>>>
>>> -----Original Message-----
>>> From: varnish-misc-bounces at varnish-cache.org
>>> [mailto:varnish-misc-bounces at varnish-cache.org] On Behalf Of George
>>> Georgovassilis
>>> Sent: Wednesday, January 05, 2011 11:18 AM
>>> To: varnish-misc at projects.linpro.no
>>> Subject: Re: Connections dropped under load
>>>
>>> Hello Cosimo,
>>>
>>> Thank you for the quick reply. After your hint I had the tests run again
>>> but couldn't detect that pattern. What susprised me though after looking
>>> through the logs is that almost all requests by the load generator
>>> complete in a timely manner (<  1 sec), but all requests generated by a
>>> real browser (IE, FF, Opera) will be served much later or even run into
>>> a timeout.
>>>
>>> On 05.01.2011 16:30, Cosimo Streppone wrote:
>>>
>>>> On Wed, 05 Jan 2011 16:20:31 +0100, George Georgovassilis
>>>> <g.georgovassilis at gmail.com>  wrote:
>>>>
>>>>  I'm having trouble with dropped connections under a loadtest.
>>>>>
>>>>> The problem: As a measure for response, I am requesting an image from
>>>>> the webapp running in Tomcat while the loadtest is underway. However
>>>>> that either times out or is delivered after several seconds. Varnishlog
>>>>> will often either not show the request (RxURL) at all, or show it
>>>>> several seconds after the browser dispatched it.
>>>>>
>>>> Hi George,
>>>>
>>>> if you measure the time you mention as "several seconds"
>>>> and it's either 3 or 9 seconds, I think what you're seeing
>>>> is a client-side TCP retransmit timeout.
>>>>
>>>> I experienced that, both under load testing,
>>>> and in real production setups.
>>>>
>>>>
>>> _______________________________________________
>>> varnish-misc mailing list
>>> varnish-misc at varnish-cache.org
>>> http://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
>>>
>>>
>>
>> _______________________________________________
>> varnish-misc mailing list
>> varnish-misc at varnish-cache.org
>> http://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
>>
>
>
>
> --
> Stig Bakken
> CTO, Zedge.net - free your phone!
>
>
>
> _______________________________________________
> varnish-misc mailing list
> varnish-misc at varnish-cache.org
> http://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
>

-- 
Stig Bakken
CTO, Zedge.net - free your phone!
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/varnish-misc/attachments/20110105/31dc6e92/attachment-0003.html>