Connections dropped under load

George Georgovassilis g.georgovassilis at gmail.com
Thu Jan 6 00:24:07 CET 2011


Thanks a lot Stig... your analysis in that discussion goes way beyond 
mind. Did you ever sort it out?

In the meantime I found an unlikely setting that solved my problem: the 
session_linger. I got it from here [1] and thought it wouldn't hurt and 
it nearly killed me. My initial tests were conducted with a value of 
150, I had to lower it to 20 to get my test through.

Thanks + best regards

[1] https://kristianlyng.wordpress.com/2009/10/19/high-end-varnish-tuning/

On 05.01.2011 23:56, Stig Bakken wrote:
> This thread: 
> http://www.varnish-cache.org/lists/pipermail/varnish-misc/2010-December/005258.html 
>
>
>  - Stig
>
> On Wed, Jan 5, 2011 at 11:44 PM, George Georgovassilis 
> <g.georgovassilis at gmail.com <mailto:g.georgovassilis at gmail.com>> wrote:
>
>     Hello Stig,
>
>     Thanks for the insight. I'm still on the logs, though not sure
>     where to start - it's not like that there are any errors in it so
>     I'm not really sure what to look for. Do you have a pointer to
>     that discussion you are referring to?
>
>
>     On 05.01.2011 23:41, Stig Bakken wrote:
>>     This seems similar to what I've been seeing, described in an
>>     earlier thread from before christmas.  In my case it was not
>>     during benchmarking, but when serving production load of around
>>     300 req/s per server.  Modern tcpip stacks on modern hardware
>>     should handle this without blinking.
>>
>>     Did you have the chance to capture the problem with varnishlog so
>>     you can replay/analyze it?
>>
>>      - Stig
>>
>>     On Wed, Jan 5, 2011 at 11:18 PM, George Georgovassilis
>>     <g.georgovassilis at gmail.com <mailto:g.georgovassilis at gmail.com>>
>>     wrote:
>>
>>         I removed the varnish instance so that the load generator is
>>         directly hitting Tomcat. Naturally, the request rate drops to
>>         70 requests/sec with a CPU load of 100%... however
>>         connections don't drop anymore, no timeouts occur and the
>>         application remains pretty responsive. To recap, these are
>>         the possible scenarios:
>>
>>         1. The networking layer is overtaxed with the original 300
>>         reqs/sec. I don't believe that, because the load generator
>>         doesn't record any dropped connections while a simple browser
>>         can't connect.
>>
>>         2. Tomcat is overtaxed. That also seems not plausible, since
>>         it is not servicing any requests under the load test - all is
>>         done by varnish. Even if, as I said when removing varnish
>>         from in between, it serves the requests just fine.
>>
>>         3. Varnish is overtaxed. Somehow that also doesn't make
>>         sense, since it is servicing the load generator just fine...
>>         but will refuse to serve browser requests.
>>
>>         4. Varnish, when under load, is picky about what connections
>>         to serve.
>>
>>         I'm stuck :-)
>>
>>
>>         On 05.01.2011 17:59, Bob Camp wrote:
>>
>>             Hi
>>
>>             Running simple load tests both on Apache directly, and on
>>             Varnish - both
>>             seem to experience "long delays" on a small percentage of
>>             the requests. The
>>             problem does not appear to happen with low loads. It does
>>             come up as CPU
>>             usage becomes an issue. It also is hard to make happen
>>             with a single stream
>>             of requests. It seems to come up much quicker with many
>>             requests done in
>>             parallel.
>>
>>             I've always *assumed* that the poor little TCP/IP hamster
>>             simply ran out of
>>             breath and started dropping connections.
>>
>>             Bob
>>
>>             -----Original Message-----
>>             From: varnish-misc-bounces at varnish-cache.org
>>             <mailto:varnish-misc-bounces at varnish-cache.org>
>>             [mailto:varnish-misc-bounces at varnish-cache.org
>>             <mailto:varnish-misc-bounces at varnish-cache.org>] On
>>             Behalf Of George
>>             Georgovassilis
>>             Sent: Wednesday, January 05, 2011 11:18 AM
>>             To: varnish-misc at projects.linpro.no
>>             <mailto:varnish-misc at projects.linpro.no>
>>             Subject: Re: Connections dropped under load
>>
>>             Hello Cosimo,
>>
>>             Thank you for the quick reply. After your hint I had the
>>             tests run again
>>             but couldn't detect that pattern. What susprised me
>>             though after looking
>>             through the logs is that almost all requests by the load
>>             generator
>>             complete in a timely manner (<  1 sec), but all requests
>>             generated by a
>>             real browser (IE, FF, Opera) will be served much later or
>>             even run into
>>             a timeout.
>>
>>             On 05.01.2011 16:30, Cosimo Streppone wrote:
>>
>>                 On Wed, 05 Jan 2011 16:20:31 +0100, George Georgovassilis
>>                 <g.georgovassilis at gmail.com
>>                 <mailto:g.georgovassilis at gmail.com>>  wrote:
>>
>>                     I'm having trouble with dropped connections under
>>                     a loadtest.
>>
>>                     The problem: As a measure for response, I am
>>                     requesting an image from
>>                     the webapp running in Tomcat while the loadtest
>>                     is underway. However
>>                     that either times out or is delivered after
>>                     several seconds. Varnishlog
>>                     will often either not show the request (RxURL) at
>>                     all, or show it
>>                     several seconds after the browser dispatched it.
>>
>>                 Hi George,
>>
>>                 if you measure the time you mention as "several seconds"
>>                 and it's either 3 or 9 seconds, I think what you're
>>                 seeing
>>                 is a client-side TCP retransmit timeout.
>>
>>                 I experienced that, both under load testing,
>>                 and in real production setups.
>>
>>
>>             _______________________________________________
>>             varnish-misc mailing list
>>             varnish-misc at varnish-cache.org
>>             <mailto:varnish-misc at varnish-cache.org>
>>             http://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
>>
>>
>>
>>         _______________________________________________
>>         varnish-misc mailing list
>>         varnish-misc at varnish-cache.org
>>         <mailto:varnish-misc at varnish-cache.org>
>>         http://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
>>
>>
>>
>>
>>     -- 
>>     Stig Bakken
>>     CTO, Zedge.net - free your phone!
>
>
>     _______________________________________________
>     varnish-misc mailing list
>     varnish-misc at varnish-cache.org <mailto:varnish-misc at varnish-cache.org>
>     http://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
>
>
>
>
> -- 
> Stig Bakken
> CTO, Zedge.net - free your phone!

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/varnish-misc/attachments/20110106/b285558d/attachment-0003.html>


More information about the varnish-misc mailing list