Hi list,<div><br></div><div>I am trying out varnish on some boxes mostly serving static files with Apache. It's almost working smoothly, but sometimes we have requests that seem to take an inordinate amount of time.</div>
<div><br></div><div>The backend servers are running with keepalives disabled (since there was a bug related to keepalives in varnish 2.1.4). I have tried running 2.1.3, 2.1.4 and yesterday's trunk, all with the same result.</div>
<div><br></div><div>I have a script processing varnishlog output to give me the serving latency based on the ReqEnd tag, and frequently see objects served from cache, size around 25kB, that take 5+ seconds to serve. At first I hypothesized that this included the time required to send enough packets to the client that the remaining data was in kernel buffers and varnish considered the request done. To test this, I wrote a test client that fired off a request to some cached object, slept 10 seconds, then read headers and body. From varnishlog's point of view, this request took >1ms to complete, so that was an incorrect assumption.</div>
<div><br></div><div>So right now I'm out of ideas. Has anyone run into this before?</div><div><br></div><div> - Stig</div><div><br></div><div>/etc/sysconfig/varnish:</div><blockquote class="webkit-indent-blockquote" style="margin: 0 0 0 40px; border: none; padding: 0px;">
<div><div>NFILES=131072</div></div><div><div>MEMLOCK=82000</div></div><div><div>DAEMON_OPTS="\</div></div><div><div> -T localhost:6082 \</div></div><div><div> -f /etc/varnish/fs.vcl \</div></div><div><div> -u varnish \</div>
</div><div><div> -g varnish \</div></div><div><div> -S /etc/varnish/secret \</div></div><div><div> -p thread_pools=8 \</div></div><div><div> -p thread_pool_add_delay=2 \</div></div><div><div> -p thread_pool_min=100 \</div>
</div><div><div> -p thread_pool_max=20000 \</div></div><div><div> -p thread_pool_timeout=300 \</div></div><div><div> -t 86400 \</div></div><div><div> -s malloc,8G"</div></div></blockquote><div><br></div><div>VCL file:</div>
<blockquote class="webkit-indent-blockquote" style="margin: 0 0 0 40px; border: none; padding: 0px;"><div><div>backend default {</div></div><div><div> .host = "127.0.0.1";</div></div><div><div> .port = "8088";</div>
</div><div><div>}</div></div><div><div><br></div></div><div><div>sub vcl_recv {</div></div><div><div> if (req.request != "GET" &&</div></div><div><div> req.request != "HEAD" &&</div>
</div><div><div> req.request != "PUT" &&</div></div><div><div> req.request != "POST" &&</div></div><div><div> req.request != "TRACE" &&</div></div>
<div><div> req.request != "OPTIONS" &&</div></div><div><div> req.request != "DELETE") {</div></div><div><div> /* Non-RFC2616 or CONNECT which is weird. */</div></div><div><div>
return (pipe);</div></div><div><div> }</div></div><div><div> if (req.restarts == 0) {</div></div><div><div> if (req.http.x-forwarded-for) {</div></div><div><div> set req.http.X-Forwarded-For =</div>
</div><div><div> req.http.X-Forwarded-For ", " client.ip;</div></div><div><div> } else {</div></div><div><div> set req.http.X-Forwarded-For = client.ip;</div></div><div><div> }</div>
</div><div><div> }</div></div><div><div><br></div></div><div><div> if (req.request != "GET") {</div></div><div><div> return (pass);</div></div><div><div> }</div></div><div><div><br></div></div><div>
<div> if (req.url ~ "-t\.(jpg|jpeg|gif|png)$"</div></div><div><div> || req.url ~ "^/content-management\.php\?(zth|zcr)="</div></div><div><div> ) {</div></div><div><div> remove req.http.Cookie;</div>
</div><div><div> return (lookup);</div></div><div><div> }</div></div><div><div><br></div></div><div><div> if (req.url ~ "^/video-streamer\.php") {</div></div><div><div> return (pipe);</div></div>
<div><div> }</div></div><div><div><br></div></div><div><div> return (pass);</div></div><div><div><br></div></div><div><div>}</div></div><div><div><br></div></div><div><div>sub vcl_pipe {</div></div><div><div> set bereq.http.Connection = "close";</div>
</div><div><div>}</div></div><div><div><br></div></div><div><div>sub vcl_fetch {</div></div><div><div> remove beresp.http.ETag;</div></div><div><div>}</div></div></blockquote><div><br></div><div><br></div><div>An in case any of this matters, sysctl stuff:</div>
<blockquote class="webkit-indent-blockquote" style="margin: 0 0 0 40px; border: none; padding: 0px;"><div><div>net.core.rmem_max=16777216</div></div><div><div>net.core.wmem_max=16777216</div></div><div><div>net.ipv4.tcp_rmem=4096 87380 16777216</div>
</div><div><div>net.ipv4.tcp_wmem=4096 65536 16777216</div></div><div><div>net.ipv4.tcp_fin_timeout = 3</div></div><div><div>net.ipv4.tcp_tw_recycle = 1</div></div><div><div>net.core.netdev_max_backlog = 30000</div></div>
<div><div>net.ipv4.tcp_no_metrics_save=1</div></div><div><div>net.core.somaxconn = 262144</div></div><div><div>net.ipv4.tcp_syncookies = 1</div></div><div><div>net.ipv4.tcp_max_orphans = 262144</div></div><div><div>net.ipv4.tcp_max_syn_backlog = 262144</div>
</div><div><div>net.ipv4.tcp_synack_retries = 2</div></div><div><div>net.ipv4.tcp_syn_retries = 2</div></div></blockquote><div><br clear="all"><br>-- <br>Stig Bakken<br>CTO, Zedge.net - free your phone!<br>
</div>