high memory usage with malloc and file backends configured

[Bender, Charles]

Hi,

There is a large discrepancy between Varnish resident memory reported by top vs reported by varnishstat. Varnish is configured with both malloc and file storage; 20G malloc and 75G file storage.

These are the startup parameters-

VARNISH_STORAGE="memcache=malloc,20G -s filecache=file,/mnt/xvdf1/varnish/varnish_storage.bin,75G"

After running for a few days top is reporting more than twice amount of memory used for varnishd process than varnishstat.

From top-

  PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND
 1083 varnish   20   0 82.235g 0.014t 7.765g S  75.7 51.4   2159:07 varnishd

From varnishstat-

SMA.memcache.g_bytes                                                       5.20G        39.65K          .            5.20G         5.20G   5.20G
SMA.memcache.g_space                                                      14.80G       -39.65K          .           14.80G        14.80G              14.80G
SMF.filecache.g_bytes                                                      7.78G        27.97K          .         7.78G         7.78G         7.78G
SMF.filecache.g_space                                                     67.22G       -27.97K          .      67.22G        67.22G        67.22G

This is the relevant part of the VCL regarding storage backend selection-

sub vcl_backend_response {
                # define separate cache storage groups
                if (bereq.http.host ~ "^(encore|thereal|static)\.(beachcamera|buydig)\.com") {
                                set beresp.storage_hint = "filecache";
                                set beresp.http.X-Cache-Storage = "disk";
                } elsif (bereq.url ~ "(?i)\.(jpg|jpeg|gif|ico|pdf|swf|png|zip)") {
                                set beresp.storage_hint = "filecache";
                                set beresp.http.X-Cache-Storage = "disk";
                } elsif (bereq.url ~ "(?i)product\-image\.aspx") {
                                set beresp.storage_hint = "filecache";
                                set beresp.http.X-Cache-Storage = "disk";
                } else {
                                set beresp.storage_hint = "memcache";
                                set beresp.http.X-Cache-Storage = "memory";
                }

Can post entire VCL if needed.

Would think that since varnishstat reports 5.20G RAM used the resident memory should be around 6-7G, 14G seems excessively high. File storage should use minimal resident memory, correct?

Varnish was installed from Varnish Cache 4.1 repo. No VMODs loaded except std and directors. Using latest 4.1.7

Anything else you need please let me know.


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/varnish-misc/attachments/20170710/21a74ad5/attachment.html>