Connections dropped under load

Wed Jan 5 23:44:21 CET 2011

Hello Stig,

Thanks for the insight. I'm still on the logs, though not sure where to 
start - it's not like that there are any errors in it so I'm not really 
sure what to look for. Do you have a pointer to that discussion you are 
referring to?

On 05.01.2011 23:41, Stig Bakken wrote:
> This seems similar to what I've been seeing, described in an earlier 
> thread from before christmas.  In my case it was not during 
> benchmarking, but when serving production load of around 300 req/s per 
> server.  Modern tcpip stacks on modern hardware should handle this 
> without blinking.
>
> Did you have the chance to capture the problem with varnishlog so you 
> can replay/analyze it?
>
>  - Stig
>
> On Wed, Jan 5, 2011 at 11:18 PM, George Georgovassilis 
> <g.georgovassilis at gmail.com <mailto:g.georgovassilis at gmail.com>> wrote:
>
>     I removed the varnish instance so that the load generator is
>     directly hitting Tomcat. Naturally, the request rate drops to 70
>     requests/sec with a CPU load of 100%... however connections don't
>     drop anymore, no timeouts occur and the application remains pretty
>     responsive. To recap, these are the possible scenarios:
>
>     1. The networking layer is overtaxed with the original 300
>     reqs/sec. I don't believe that, because the load generator doesn't
>     record any dropped connections while a simple browser can't connect.
>
>     2. Tomcat is overtaxed. That also seems not plausible, since it is
>     not servicing any requests under the load test - all is done by
>     varnish. Even if, as I said when removing varnish from in between,
>     it serves the requests just fine.
>
>     3. Varnish is overtaxed. Somehow that also doesn't make sense,
>     since it is servicing the load generator just fine... but will
>     refuse to serve browser requests.
>
>     4. Varnish, when under load, is picky about what connections to serve.
>
>     I'm stuck :-)
>
>
>     On 05.01.2011 17:59, Bob Camp wrote:
>
>         Hi
>
>         Running simple load tests both on Apache directly, and on
>         Varnish - both
>         seem to experience "long delays" on a small percentage of the
>         requests. The
>         problem does not appear to happen with low loads. It does come
>         up as CPU
>         usage becomes an issue. It also is hard to make happen with a
>         single stream
>         of requests. It seems to come up much quicker with many
>         requests done in
>         parallel.
>
>         I've always *assumed* that the poor little TCP/IP hamster
>         simply ran out of
>         breath and started dropping connections.
>
>         Bob
>
>         -----Original Message-----
>         From: varnish-misc-bounces at varnish-cache.org
>         <mailto:varnish-misc-bounces at varnish-cache.org>
>         [mailto:varnish-misc-bounces at varnish-cache.org
>         <mailto:varnish-misc-bounces at varnish-cache.org>] On Behalf Of
>         George
>         Georgovassilis
>         Sent: Wednesday, January 05, 2011 11:18 AM
>         To: varnish-misc at projects.linpro.no
>         <mailto:varnish-misc at projects.linpro.no>
>         Subject: Re: Connections dropped under load
>
>         Hello Cosimo,
>
>         Thank you for the quick reply. After your hint I had the tests
>         run again
>         but couldn't detect that pattern. What susprised me though
>         after looking
>         through the logs is that almost all requests by the load generator
>         complete in a timely manner (<  1 sec), but all requests
>         generated by a
>         real browser (IE, FF, Opera) will be served much later or even
>         run into
>         a timeout.
>
>         On 05.01.2011 16:30, Cosimo Streppone wrote:
>
>             On Wed, 05 Jan 2011 16:20:31 +0100, George Georgovassilis
>             <g.georgovassilis at gmail.com
>             <mailto:g.georgovassilis at gmail.com>>  wrote:
>
>                 I'm having trouble with dropped connections under a
>                 loadtest.
>
>                 The problem: As a measure for response, I am
>                 requesting an image from
>                 the webapp running in Tomcat while the loadtest is
>                 underway. However
>                 that either times out or is delivered after several
>                 seconds. Varnishlog
>                 will often either not show the request (RxURL) at all,
>                 or show it
>                 several seconds after the browser dispatched it.
>
>             Hi George,
>
>             if you measure the time you mention as "several seconds"
>             and it's either 3 or 9 seconds, I think what you're seeing
>             is a client-side TCP retransmit timeout.
>
>             I experienced that, both under load testing,
>             and in real production setups.
>
>
>         _______________________________________________
>         varnish-misc mailing list
>         varnish-misc at varnish-cache.org
>         <mailto:varnish-misc at varnish-cache.org>
>         http://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
>
>
>
>     _______________________________________________
>     varnish-misc mailing list
>     varnish-misc at varnish-cache.org <mailto:varnish-misc at varnish-cache.org>
>     http://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
>
>
>
>
> -- 
> Stig Bakken
> CTO, Zedge.net - free your phone!

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/varnish-misc/attachments/20110105/0d08832f/attachment-0003.html>