high memory usage with malloc and file backends configured

[Guillaume Quintard]

Hi Charles,

So, if I'm reading this right, there's no discrepancy. Varnish will malloc
the full storage (malloc), and will mmap a file the size of the full
storage (file). So even though the storage is not used, it's allocated.

Does it make sense, or did I miss something?

-- 
Guillaume Quintard

On Mon, Jul 10, 2017 at 1:23 PM, Bender, Charles <charles at beachcamera.com>
wrote:

> Hi,
>
>
>
> There is a large discrepancy between Varnish resident memory reported by
> top vs reported by varnishstat. Varnish is configured with both malloc and
> file storage; 20G malloc and 75G file storage.
>
>
>
> These are the startup parameters-
>
>
>
> VARNISH_STORAGE="memcache=malloc,20G -s filecache=file,/mnt/xvdf1/
> varnish/varnish_storage.bin,75G"
>
>
>
> After running for a few days top is reporting more than twice amount of
> memory used for varnishd process than varnishstat.
>
>
>
> From top-
>
>
>
>   PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+
> COMMAND
>
>
>  1083 varnish   20   0 82.235g 0.014t 7.765g S  75.7 51.4   2159:07
> varnishd
>
>
>
>
> From varnishstat-
>
>
>
> SMA.memcache.g_bytes
> 5.20G        39.65K          .            5.20G         5.20G   5.20G
>
> SMA.memcache.g_space
> 14.80G       -39.65K          .           14.80G        14.80G
> 14.80G
>
> SMF.filecache.g_bytes
>       7.78G        27.97K          .         7.78G         7.78G
> 7.78G
>
> SMF.filecache.g_space
> 67.22G       -27.97K          .      67.22G        67.22G        67.22G
>
>
>
> This is the relevant part of the VCL regarding storage backend selection-
>
>
>
> sub vcl_backend_response {
>
>                 # define separate cache storage groups
>
>                 if (bereq.http.host ~ "^(encore|thereal|static)\.(beachcamera|buydig)\.com")
> {
>
>                                 set beresp.storage_hint = "filecache";
>
>                                 set beresp.http.X-Cache-Storage = "disk";
>
>                 } elsif (bereq.url ~ "(?i)\.(jpg|jpeg|gif|ico|pdf|swf|png|zip)")
> {
>
>                                 set beresp.storage_hint = "filecache";
>
>                                 set beresp.http.X-Cache-Storage = "disk";
>
>                 } elsif (bereq.url ~ "(?i)product\-image\.aspx") {
>
>                                 set beresp.storage_hint = "filecache";
>
>                                 set beresp.http.X-Cache-Storage = "disk";
>
>                 } else {
>
>                                 set beresp.storage_hint = "memcache";
>
>                                 set beresp.http.X-Cache-Storage =
> "memory";
>
>                 }
>
>
>
> Can post entire VCL if needed.
>
>
>
> Would think that since varnishstat reports 5.20G RAM used the resident
> memory should be around 6-7G, 14G seems excessively high. File storage
> should use minimal resident memory, correct?
>
>
>
> Varnish was installed from Varnish Cache 4.1 repo. No VMODs loaded except
> std and directors. Using latest 4.1.7
>
>
>
> Anything else you need please let me know.
>
>
>
>
>
> _______________________________________________
> varnish-misc mailing list
> varnish-misc at varnish-cache.org
> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/varnish-misc/attachments/20170710/fe235aba/attachment-0001.html>