<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN">
<html>
  <head>
    <meta content="text/html; charset=UTF-8" http-equiv="Content-Type">
  </head>
  <body bgcolor="#ffffff" text="#000000">
    Hello Stig,<br>
    <br>
    Thanks for the insight. I'm still on the logs, though not sure where
    to start - it's not like that there are any errors in it so I'm not
    really sure what to look for. Do you have a pointer to that
    discussion you are referring to?<br>
    <br>
    On 05.01.2011 23:41, Stig Bakken wrote:
    <blockquote
      cite="mid:AANLkTim3zep1q3sOs52iwNefYpNrHYEv1Wg4K0xYGW2G@mail.gmail.com"
      type="cite">This seems similar to what I've been seeing, described
      in an earlier thread from before christmas.  In my case it was not
      during benchmarking, but when serving production load of around
      300 req/s per server.  Modern tcpip stacks on modern hardware
      should handle this without blinking.
      <div>
        <br>
      </div>
      <div>Did you have the chance to capture the problem with
        varnishlog so you can replay/analyze it?</div>
      <div><br>
      </div>
      <div> - Stig<br>
        <br>
        <div class="gmail_quote">On Wed, Jan 5, 2011 at 11:18 PM, George
          Georgovassilis <span dir="ltr"><<a moz-do-not-send="true"
              href="mailto:g.georgovassilis@gmail.com">g.georgovassilis@gmail.com</a>></span>
          wrote:<br>
          <blockquote class="gmail_quote" style="margin: 0pt 0pt 0pt
            0.8ex; border-left: 1px solid rgb(204, 204, 204);
            padding-left: 1ex;">I removed the varnish instance so that
            the load generator is directly hitting Tomcat. Naturally,
            the request rate drops to 70 requests/sec with a CPU load of
            100%... however connections don't drop anymore, no timeouts
            occur and the application remains pretty responsive. To
            recap, these are the possible scenarios:<br>
            <br>
            1. The networking layer is overtaxed with the original 300
            reqs/sec. I don't believe that, because the load generator
            doesn't record any dropped connections while a simple
            browser can't connect.<br>
            <br>
            2. Tomcat is overtaxed. That also seems not plausible, since
            it is not servicing any requests under the load test - all
            is done by varnish. Even if, as I said when removing varnish
            from in between, it serves the requests just fine.<br>
            <br>
            3. Varnish is overtaxed. Somehow that also doesn't make
            sense, since it is servicing the load generator just fine...
            but will refuse to serve browser requests.<br>
            <br>
            4. Varnish, when under load, is picky about what connections
            to serve.<br>
            <br>
            I'm stuck :-)
            <div>
              <div class="h5"><br>
                <br>
                On 05.01.2011 17:59, Bob Camp wrote:<br>
                <blockquote class="gmail_quote" style="margin: 0pt 0pt
                  0pt 0.8ex; border-left: 1px solid rgb(204, 204, 204);
                  padding-left: 1ex;">
                  Hi<br>
                  <br>
                  Running simple load tests both on Apache directly, and
                  on Varnish - both<br>
                  seem to experience "long delays" on a small percentage
                  of the requests. The<br>
                  problem does not appear to happen with low loads. It
                  does come up as CPU<br>
                  usage becomes an issue. It also is hard to make happen
                  with a single stream<br>
                  of requests. It seems to come up much quicker with
                  many requests done in<br>
                  parallel.<br>
                  <br>
                  I've always *assumed* that the poor little TCP/IP
                  hamster simply ran out of<br>
                  breath and started dropping connections.<br>
                  <br>
                  Bob<br>
                  <br>
                  -----Original Message-----<br>
                  From: <a moz-do-not-send="true"
                    href="mailto:varnish-misc-bounces@varnish-cache.org"
                    target="_blank">varnish-misc-bounces@varnish-cache.org</a><br>
                  [mailto:<a moz-do-not-send="true"
                    href="mailto:varnish-misc-bounces@varnish-cache.org"
                    target="_blank">varnish-misc-bounces@varnish-cache.org</a>]
                  On Behalf Of George<br>
                  Georgovassilis<br>
                  Sent: Wednesday, January 05, 2011 11:18 AM<br>
                  To: <a moz-do-not-send="true"
                    href="mailto:varnish-misc@projects.linpro.no"
                    target="_blank">varnish-misc@projects.linpro.no</a><br>
                  Subject: Re: Connections dropped under load<br>
                  <br>
                  Hello Cosimo,<br>
                  <br>
                  Thank you for the quick reply. After your hint I had
                  the tests run again<br>
                  but couldn't detect that pattern. What susprised me
                  though after looking<br>
                  through the logs is that almost all requests by the
                  load generator<br>
                  complete in a timely manner (<  1 sec), but all
                  requests generated by a<br>
                  real browser (IE, FF, Opera) will be served much later
                  or even run into<br>
                  a timeout.<br>
                  <br>
                  On 05.01.2011 16:30, Cosimo Streppone wrote:<br>
                  <blockquote class="gmail_quote" style="margin: 0pt 0pt
                    0pt 0.8ex; border-left: 1px solid rgb(204, 204,
                    204); padding-left: 1ex;">
                    On Wed, 05 Jan 2011 16:20:31 +0100, George
                    Georgovassilis<br>
                    <<a moz-do-not-send="true"
                      href="mailto:g.georgovassilis@gmail.com"
                      target="_blank">g.georgovassilis@gmail.com</a>>
                     wrote:<br>
                    <br>
                    <blockquote class="gmail_quote" style="margin: 0pt
                      0pt 0pt 0.8ex; border-left: 1px solid rgb(204,
                      204, 204); padding-left: 1ex;">
                      I'm having trouble with dropped connections under
                      a loadtest.<br>
                      <br>
                      The problem: As a measure for response, I am
                      requesting an image from<br>
                      the webapp running in Tomcat while the loadtest is
                      underway. However<br>
                      that either times out or is delivered after
                      several seconds. Varnishlog<br>
                      will often either not show the request (RxURL) at
                      all, or show it<br>
                      several seconds after the browser dispatched it.<br>
                    </blockquote>
                    Hi George,<br>
                    <br>
                    if you measure the time you mention as "several
                    seconds"<br>
                    and it's either 3 or 9 seconds, I think what you're
                    seeing<br>
                    is a client-side TCP retransmit timeout.<br>
                    <br>
                    I experienced that, both under load testing,<br>
                    and in real production setups.<br>
                    <br>
                  </blockquote>
                  <br>
                  _______________________________________________<br>
                  varnish-misc mailing list<br>
                  <a moz-do-not-send="true"
                    href="mailto:varnish-misc@varnish-cache.org"
                    target="_blank">varnish-misc@varnish-cache.org</a><br>
                  <a moz-do-not-send="true"
                    href="http://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc"
                    target="_blank">http://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc</a><br>
                  <br>
                </blockquote>
                <br>
                <br>
                _______________________________________________<br>
                varnish-misc mailing list<br>
                <a moz-do-not-send="true"
                  href="mailto:varnish-misc@varnish-cache.org"
                  target="_blank">varnish-misc@varnish-cache.org</a><br>
                <a moz-do-not-send="true"
                  href="http://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc"
                  target="_blank">http://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc</a><br>
              </div>
            </div>
          </blockquote>
        </div>
        <br>
        <br clear="all">
        <br>
        -- <br>
        Stig Bakken<br>
        CTO, Zedge.net - free your phone!<br>
      </div>
    </blockquote>
    <br>
  </body>
</html>