Connections dropped under load
George Georgovassilis
g.georgovassilis at gmail.com
Thu Jan 6 00:24:07 CET 2011
Thanks a lot Stig... your analysis in that discussion goes way beyond
mind. Did you ever sort it out?
In the meantime I found an unlikely setting that solved my problem: the
session_linger. I got it from here [1] and thought it wouldn't hurt and
it nearly killed me. My initial tests were conducted with a value of
150, I had to lower it to 20 to get my test through.
Thanks + best regards
[1] https://kristianlyng.wordpress.com/2009/10/19/high-end-varnish-tuning/
On 05.01.2011 23:56, Stig Bakken wrote:
> This thread:
> http://www.varnish-cache.org/lists/pipermail/varnish-misc/2010-December/005258.html
>
>
> - Stig
>
> On Wed, Jan 5, 2011 at 11:44 PM, George Georgovassilis
> <g.georgovassilis at gmail.com <mailto:g.georgovassilis at gmail.com>> wrote:
>
> Hello Stig,
>
> Thanks for the insight. I'm still on the logs, though not sure
> where to start - it's not like that there are any errors in it so
> I'm not really sure what to look for. Do you have a pointer to
> that discussion you are referring to?
>
>
> On 05.01.2011 23:41, Stig Bakken wrote:
>> This seems similar to what I've been seeing, described in an
>> earlier thread from before christmas. In my case it was not
>> during benchmarking, but when serving production load of around
>> 300 req/s per server. Modern tcpip stacks on modern hardware
>> should handle this without blinking.
>>
>> Did you have the chance to capture the problem with varnishlog so
>> you can replay/analyze it?
>>
>> - Stig
>>
>> On Wed, Jan 5, 2011 at 11:18 PM, George Georgovassilis
>> <g.georgovassilis at gmail.com <mailto:g.georgovassilis at gmail.com>>
>> wrote:
>>
>> I removed the varnish instance so that the load generator is
>> directly hitting Tomcat. Naturally, the request rate drops to
>> 70 requests/sec with a CPU load of 100%... however
>> connections don't drop anymore, no timeouts occur and the
>> application remains pretty responsive. To recap, these are
>> the possible scenarios:
>>
>> 1. The networking layer is overtaxed with the original 300
>> reqs/sec. I don't believe that, because the load generator
>> doesn't record any dropped connections while a simple browser
>> can't connect.
>>
>> 2. Tomcat is overtaxed. That also seems not plausible, since
>> it is not servicing any requests under the load test - all is
>> done by varnish. Even if, as I said when removing varnish
>> from in between, it serves the requests just fine.
>>
>> 3. Varnish is overtaxed. Somehow that also doesn't make
>> sense, since it is servicing the load generator just fine...
>> but will refuse to serve browser requests.
>>
>> 4. Varnish, when under load, is picky about what connections
>> to serve.
>>
>> I'm stuck :-)
>>
>>
>> On 05.01.2011 17:59, Bob Camp wrote:
>>
>> Hi
>>
>> Running simple load tests both on Apache directly, and on
>> Varnish - both
>> seem to experience "long delays" on a small percentage of
>> the requests. The
>> problem does not appear to happen with low loads. It does
>> come up as CPU
>> usage becomes an issue. It also is hard to make happen
>> with a single stream
>> of requests. It seems to come up much quicker with many
>> requests done in
>> parallel.
>>
>> I've always *assumed* that the poor little TCP/IP hamster
>> simply ran out of
>> breath and started dropping connections.
>>
>> Bob
>>
>> -----Original Message-----
>> From: varnish-misc-bounces at varnish-cache.org
>> <mailto:varnish-misc-bounces at varnish-cache.org>
>> [mailto:varnish-misc-bounces at varnish-cache.org
>> <mailto:varnish-misc-bounces at varnish-cache.org>] On
>> Behalf Of George
>> Georgovassilis
>> Sent: Wednesday, January 05, 2011 11:18 AM
>> To: varnish-misc at projects.linpro.no
>> <mailto:varnish-misc at projects.linpro.no>
>> Subject: Re: Connections dropped under load
>>
>> Hello Cosimo,
>>
>> Thank you for the quick reply. After your hint I had the
>> tests run again
>> but couldn't detect that pattern. What susprised me
>> though after looking
>> through the logs is that almost all requests by the load
>> generator
>> complete in a timely manner (< 1 sec), but all requests
>> generated by a
>> real browser (IE, FF, Opera) will be served much later or
>> even run into
>> a timeout.
>>
>> On 05.01.2011 16:30, Cosimo Streppone wrote:
>>
>> On Wed, 05 Jan 2011 16:20:31 +0100, George Georgovassilis
>> <g.georgovassilis at gmail.com
>> <mailto:g.georgovassilis at gmail.com>> wrote:
>>
>> I'm having trouble with dropped connections under
>> a loadtest.
>>
>> The problem: As a measure for response, I am
>> requesting an image from
>> the webapp running in Tomcat while the loadtest
>> is underway. However
>> that either times out or is delivered after
>> several seconds. Varnishlog
>> will often either not show the request (RxURL) at
>> all, or show it
>> several seconds after the browser dispatched it.
>>
>> Hi George,
>>
>> if you measure the time you mention as "several seconds"
>> and it's either 3 or 9 seconds, I think what you're
>> seeing
>> is a client-side TCP retransmit timeout.
>>
>> I experienced that, both under load testing,
>> and in real production setups.
>>
>>
>> _______________________________________________
>> varnish-misc mailing list
>> varnish-misc at varnish-cache.org
>> <mailto:varnish-misc at varnish-cache.org>
>> http://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
>>
>>
>>
>> _______________________________________________
>> varnish-misc mailing list
>> varnish-misc at varnish-cache.org
>> <mailto:varnish-misc at varnish-cache.org>
>> http://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
>>
>>
>>
>>
>> --
>> Stig Bakken
>> CTO, Zedge.net - free your phone!
>
>
> _______________________________________________
> varnish-misc mailing list
> varnish-misc at varnish-cache.org <mailto:varnish-misc at varnish-cache.org>
> http://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
>
>
>
>
> --
> Stig Bakken
> CTO, Zedge.net - free your phone!
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/varnish-misc/attachments/20110106/b285558d/attachment-0003.html>
More information about the varnish-misc
mailing list