Varnish Memory usage increase

[Pinakee BIswas]

I agree but the reason being we have other applications like DB servers, 
backend server, cache server etc running on the same machine (as our 
scale is not that big). Hence, memory is a sought after resource and 
needs to be optimized as much as possible.

So, if you suggest malloc would be a better storage still, I can change 
the storage to the same.

On 07/02/18 7:09 pm, Guillaume Quintard wrote:
> So, it's probably not related to your issue, but it seems your cached 
> data easily fits in memory, why use the file storage?
>
> -- 
> Guillaume Quintard
>
> On Feb 7, 2018 14:35, "Pinakee BIswas" <pinakee at waltzz.com 
> <mailto:pinakee at waltzz.com>> wrote:
>
>     Following is the data from varnishstat:
>
>     SMA.Transient.g_bytes 15.76K        -2.19K          .          
>     18.53K 15.50K        15.24K
>
>     SMF.s0.g_bytes 673.26M        47.95K          .          672.87M
>     669.89M       668.90M
>
>
>     On 07/02/18 6:51 pm, Guillaume Quintard wrote:
>>     Amount of storage used is just the sum of all the g_bytes fields
>>
>>     -- 
>>     Guillaume Quintard
>>
>>     On Feb 7, 2018 14:12, "Pinakee BIswas" <pinakee at waltzz.com
>>     <mailto:pinakee at waltzz.com>> wrote:
>>
>>         Hi Guillaume,
>>
>>         Thanks for your response and details.
>>
>>         Please find my comments inline:
>>
>>
>>         On 07/02/18 6:21 pm, Guillaume Quintard wrote:
>>>         Hi there,
>>>
>>>         So, varnish memory usage will mostly come from three directions:
>>>         - storage, include Transient, so check the g_bytes fields in
>>>         varnishstat. Passes and shortlived objects will use
>>>         Transient, so you can either reduce those, or limit the
>>>         Transient storage (unbounded by defaukt).
>>         Checked the g_bytes for transient in varnishstat. It's in Kbs.
>>>         - thread workspaces, one thread typically uses one
>>>         workspace, so you can limit the number of threads, or reduce
>>>         the workspace size
>>>         - memory fragmentation: jemalloc will fragment up to 25%,
>>>         not much you can do here.
>>>
>>>         And no, no way to modify the mmap without restarting varnish.
>>>
>>>         May I ask how much data you are caching?
>>         Could you please let me know how to figure out the amount of
>>         data cached?
>>>
>>>         -- 
>>>         Guillaume Quintard
>>>
>>>         On Wed, Feb 7, 2018 at 1:34 PM, Pinakee BIswas
>>>         <pinakee at waltzz.com <mailto:pinakee at waltzz.com>> wrote:
>>>
>>>             Hi,
>>>
>>>             We have been using varnish since more than a year for
>>>             our ecommerce site.
>>>
>>>             Current version is 4.1.8.
>>>
>>>             OS is Linux. RAM is 8GB.
>>>
>>>             I am observing that the varnish memory usage is
>>>             increasing (right now 7%) and so is resident memory
>>>             usage increasing.
>>>
>>>             The storage being used is file:
>>>
>>>             -s file,/tmp/varnish/,${storage}
>>>
>>>             I am not sure about the reason for the increase in
>>>             varnish memory usage:
>>>
>>>               * Is there a way to limit the varnish memory usage?
>>>               * How can I diagnose what is consuming memory?
>>>               * Most of our web pages have maximum 2 days of cache.
>>>                 Also, some of the pages might be least visited.
>>>               * Is there a way to manipulate varnish mmap (when file
>>>                 storage is used)?
>>>
>>>             Would appreciate any help on the above for efficient use
>>>             of varnish.
>>>
>>>             Thanks,
>>>
>>>             Pinakee
>>>
>>>
>>>
>>>             _______________________________________________
>>>             varnish-misc mailing list
>>>             varnish-misc at varnish-cache.org
>>>             <mailto:varnish-misc at varnish-cache.org>
>>>             https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
>>>             <https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc>
>>>
>>>
>>
>

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/varnish-misc/attachments/20180207/e14cd0f2/attachment-0001.html>