Performance issue with Varnish

[Krishna Kumar (Engineering)]

Hi all,

I am testing using Varnish 4.0.3 in our datacenter. The test setup has 24
servers (48
core, 128GB, Ubuntu 14.10, Intel ixgbe) running HAProxy, with thousand
backends,
each of which are VM's created on different servers. The HAProxy servers
all run
either HAProxy only, or HAProxy + Varnish only, the two testing
configurations, with no
other load.

Model 1 (Direct):      Clients   -->   [ HAProxy ]   -->   2300 Backends
Model 2 (Caching):  Clients   -->  [ HAProxy   -->   1 Varnish backend ]
-->  2300 Backends

The Square brackets above indicate that this item runs on a single
baremetal as described
above. Diagrammatically:
                                         Clients (1000 VM's)

________    _______     ________   ________     (24 baremetals, each
|HAProxy|    |HAProxy|    |HAProxy|    |HAProxy|       of which runs either
|Varnish  |    |Varnish  |    |Varnish   |    |Varnish  |       HAProxy or
HAProxy+Varnish)
--------------   --------------    --------------    --------------
       |                    |                    |                    |
       v                   v                  v                    v
B1-B2300     B1-B2300   B1-B2300     B1-B2300    (2300 Backends, VM's
running nginx)

For the Caching setup, Varnish is configured to run on each baremetal on the
'lo' interface (on the same system). The test is admittedly primitive, each
of the
2300 clients simply gets the same 128 byte or 128K byte for a 15 minute run,
using wrk with "-t100 -c800". The following table shows the RPS and BW for
128
bytes and 128K bytes, for both "Direct" and "Caching" scenarios, and the %
change:

|--------|--------------------------------------|-------------------------------------|---------------------------------|
| I/O    |                Req Per Sec         |
BW                           |            Load                    |
| Size  |  Direct       Cache          %   |   Direct      Cache
%         |  Direct     Cache     %    |
|--------|--------------------------------------|--------------------------------------|---------------------------------|
| 128   | 5559603  2112749   -62% |  1770      780        -56%     |  629
   197       -69%      |
| 128K |  118294   138990      18% | 15158    17429    15%       | 143
389        172%     |
|---------|--------------------------------------|-------------------------------------|---------------------------------|

(hope it is readable)

For small packets, Varnish gets much lower RPS and BW (about half), but
also takes
lesser system load (measured by top). For large packets, Varnish gets about
15-18%
increase in RPS and BW, but significantly increased system load.
Varnishstat from a
typical server:
       MAIN.cache_hit              73512277     14655.56 Cache hits
       MAIN.cache_miss                    2               0.00 Cache misses

/etc/default/varnish has:

NFILES=262144
MEMLOCK=82000
DAEMON_OPTS="-a :6081 -T localhost:6082 -f /etc/varnish/default.vcl \
         -p thread_pools=2 -p thread_pool_min=500 -p thread_pool_max=750 \
         -p thread_pool_add_delay=2 -S /etc/varnish/secret -s malloc,64G"

/etc/varnish/default.vcl is created with the 2300 VM ip addresses as:

backend varnish_server_1 {
    .host = "10.32.118.14";
    .port = "80";
    .connect_timeout = 5s;
    .first_byte_timeout = 5s;
    .between_bytes_timeout = 2s;
}
...
(and so on till varnish_server_2300)

sub vcl_init {
    new varnish_cluster = directors.round_robin();
    varnish_cluster.add_backend(varnish_server_1);
    varnish_cluster.add_backend(varnish_server_2);
    ...
    varnish_cluster.add_backend(varnish_server_2300);
}

sub vcl_recv {
    set req.backend_hint = varnish_cluster.backend();
}

Could someone suggest any improvement in my setup for better performance?

Thanks,
Krishna Kumar

-- 


------------------------------------------------------------------------------------------------------------------------------------------

This email and any files transmitted with it are confidential and intended 
solely for the use of the individual or entity to whom they are addressed. 
If you have received this email in error please notify the system manager. 
This message contains confidential information and is intended only for the 
individual named. If you are not the named addressee you should not 
disseminate, distribute or copy this e-mail. Please notify the sender 
immediately by e-mail if you have received this e-mail by mistake and 
delete this e-mail from your system. If you are not the intended recipient 
you are notified that disclosing, copying, distributing or taking any 
action in reliance on the contents of this information is strictly 
prohibited. Although Flipkart has taken reasonable precautions to ensure no 
viruses are present in this email, the company cannot accept responsibility 
for any loss or damage arising from the use of this email or attachments
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/varnish-misc/attachments/20150721/1dbce58c/attachment.html>