hanging connections

[Stig Bakken]

Hi list,

I am trying out varnish on some boxes mostly serving static files with
Apache.  It's almost working smoothly, but sometimes we have requests that
seem to take an inordinate amount of time.

The backend servers are running with keepalives disabled (since there was a
bug related to keepalives in varnish 2.1.4).  I have tried running 2.1.3,
2.1.4 and yesterday's trunk, all with the same result.

I have a script processing varnishlog output to give me the serving latency
based on the ReqEnd tag, and frequently see objects served from cache, size
around 25kB, that take 5+ seconds to serve.  At first I hypothesized that
this included the time required to send enough packets to the client that
the remaining data was in kernel buffers and varnish considered the request
done.  To test this, I wrote a test client that fired off a request to some
cached object, slept 10 seconds, then read headers and body.  From
varnishlog's point of view, this request took >1ms to complete, so that was
an incorrect assumption.

So right now I'm out of ideas.  Has anyone run into this before?

 - Stig

/etc/sysconfig/varnish:

NFILES=131072
MEMLOCK=82000
DAEMON_OPTS="\
  -T localhost:6082 \
  -f /etc/varnish/fs.vcl \
  -u varnish \
  -g varnish \
  -S /etc/varnish/secret \
  -p thread_pools=8 \
  -p thread_pool_add_delay=2 \
  -p thread_pool_min=100 \
  -p thread_pool_max=20000 \
  -p thread_pool_timeout=300 \
  -t 86400 \
  -s malloc,8G"


VCL file:

backend default {
    .host = "127.0.0.1";
    .port = "8088";
}

sub vcl_recv {
    if (req.request != "GET" &&
        req.request != "HEAD" &&
        req.request != "PUT" &&
        req.request != "POST" &&
        req.request != "TRACE" &&
        req.request != "OPTIONS" &&
        req.request != "DELETE") {
        /* Non-RFC2616 or CONNECT which is weird. */
        return (pipe);
    }
    if (req.restarts == 0) {
        if (req.http.x-forwarded-for) {
            set req.http.X-Forwarded-For =
                req.http.X-Forwarded-For ", " client.ip;
        } else {
            set req.http.X-Forwarded-For = client.ip;
        }
    }

    if (req.request != "GET") {
        return (pass);
    }

    if (req.url ~ "-t\.(jpg|jpeg|gif|png)$"
        || req.url ~ "^/content-management\.php\?(zth|zcr)="
        ) {
        remove req.http.Cookie;
        return (lookup);
    }

    if (req.url ~ "^/video-streamer\.php") {
        return (pipe);
    }

    return (pass);

}

sub vcl_pipe {
    set bereq.http.Connection = "close";
}

sub vcl_fetch {
    remove beresp.http.ETag;
}



An in case any of this matters, sysctl stuff:

net.core.rmem_max=16777216
net.core.wmem_max=16777216
net.ipv4.tcp_rmem=4096 87380 16777216
net.ipv4.tcp_wmem=4096 65536 16777216
net.ipv4.tcp_fin_timeout = 3
net.ipv4.tcp_tw_recycle = 1
net.core.netdev_max_backlog = 30000
net.ipv4.tcp_no_metrics_save=1
net.core.somaxconn = 262144
net.ipv4.tcp_syncookies = 1
net.ipv4.tcp_max_orphans = 262144
net.ipv4.tcp_max_syn_backlog = 262144
net.ipv4.tcp_synack_retries = 2
net.ipv4.tcp_syn_retries = 2



-- 
Stig Bakken
CTO, Zedge.net - free your phone!
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.varnish-cache.org/lists/pipermail/varnish-misc/attachments/20101217/ec4cf20c/attachment.html>