[Varnish] #1798: Varnish requests painfully slow with large files

[Varnish]

#1798: Varnish requests painfully slow with large files
---------------------+----------------------
 Reporter:  nbetham  |       Type:  defect
   Status:  new      |   Priority:  normal
Milestone:           |  Component:  varnishd
  Version:  4.0.3    |   Severity:  normal
 Keywords:           |
---------------------+----------------------
 Hi, I have a Varnish instance running on a server which is being used to
 host some small and large-ish files. The file them selves range from 300KB
 to 20GB. I have Varnish setup to use malloc for storage. The server has
 170GB of RAM and Varnish is configured to use 160GB for malloc. Currently
 we have Apache running on the same server as the backend for Varnish. The
 server is connected to the network with a single 10Gb ethernet link. It's
 currently running Varnish 4.0.3, Ubuntu 14.04, Kernel 3.13.0-65-generic.

 When a request comes in for a small file the request completes normally
 and quickly. However when a request comes in for a large file the
 connection starts at the local max speed, 1Gbps, and then quickly ramps
 down to 40Mbps after about 1GB of the file has been downloaded. Meanwhile
 on the server one of the worker threads is stuck at 100% CPU utilization.
 Subsequent requests for the same file then complete normally, and at full
 speed throughout the request, afterwards. For every new large file
 requested a single worker thread will get stuck at 100% CPU.

 Here's the VCL running on the server. I stripped it down to the bare
 minimum to see if anything I had added was causing problems. I have also
 disabled streaming just to see if Apache was responding slowly, but that
 is not the case given that most of the large files load from Apache into
 Varnish within a few seconds.

 ===== /etc/varnish/default.vcl =====
 {{{
 #
 # This is an example VCL file for Varnish.
 #
 # It does not do anything by default, delegating control to the
 # builtin VCL. The builtin VCL is called when there is no explicit
 # return statement.
 #
 # See the VCL chapters in the Users Guide at https://www.varnish-
 cache.org/docs/
 # and http://varnish-cache.org/trac/wiki/VCLExamples for more examples.

 # Marker to tell the VCL compiler that this VCL has been adapted to the
 # new 4.0 format.
 vcl 4.0;

 # Default backend definition. Set this to point to your content server.
 backend default {
     .host = "127.0.0.1";
     .port = "7080";
 }

 sub vcl_recv {
     # Happens before we check if we have this in cache already.
     #
     # Typically you clean up the request here, removing cookies you don't
 need,
     # rewriting the request, etc.
 }

 sub vcl_backend_response {
     # Happens after we have read the response headers from the backend.
     #
     # Here you clean the response headers, removing silly Set-Cookie
 headers
     # and other mistakes your backend does.
     set beresp.ttl = 24 h;
     set beresp.do_stream = false;
     return(deliver);
 }

 sub vcl_deliver {
     # Happens when we have all the pieces we need, and are about to send
 the
     # response to the client.
     #
     # You can do accounting or modifying the final object here.
 }
 }}}

 ===== /etc/default/varnish =====
 {{{
 # Configuration file for varnish
 #
 # /etc/init.d/varnish expects the variables $DAEMON_OPTS, $NFILES and
 $MEMLOCK
 # to be set from this shell script fragment.
 #

 # Should we start varnishd at boot?  Set to "no" to disable.
 START=yes

 # Maximum number of open files (for ulimit -n)
 NFILES=131072

 # Maximum locked memory size (for ulimit -l)
 # Used for locking the shared memory log in memory.  If you increase log
 size,
 # you need to increase this number as well
 MEMLOCK=82000

 DAEMON_OPTS="-a :6080 \
              -T localhost:6082 \
              -f /etc/varnish/default.vcl \
              -S /etc/varnish/secret \
              -s malloc,160GB \
              -p send_timeout=864000"
 }}}

 Most of what I have read about Varnish is that it should be able to handle
 large files. Is there something I need to set in order to prevent worker
 threads from hanging on these large files? It seems like there might be
 something broken in the system, but i'm not sure. Let me know if you need
 more info, I would be happy to provide it. Any help with this would be
 awesome!

 Thanks,
 Neil

-- 
Ticket URL: <https://www.varnish-cache.org/trac/ticket/1798>
Varnish <https://varnish-cache.org/>
The Varnish HTTP Accelerator