Varnish Memory usage increase

[Pinakee BIswas]

Hi,

Could you please suggest what would be the optimal configuration for 
varnishd in my scenario - in terms of memory usage and performance?

Thanks,

Pinakee


On 07/02/18 7:19 pm, Pinakee BIswas wrote:
>
> I agree but the reason being we have other applications like DB 
> servers, backend server, cache server etc running on the same machine 
> (as our scale is not that big). Hence, memory is a sought after 
> resource and needs to be optimized as much as possible.
>
> So, if you suggest malloc would be a better storage still, I can 
> change the storage to the same.
>
> On 07/02/18 7:09 pm, Guillaume Quintard wrote:
>> So, it's probably not related to your issue, but it seems your cached 
>> data easily fits in memory, why use the file storage?
>>
>> -- 
>> Guillaume Quintard
>>
>> On Feb 7, 2018 14:35, "Pinakee BIswas" <pinakee at waltzz.com 
>> <mailto:pinakee at waltzz.com>> wrote:
>>
>>     Following is the data from varnishstat:
>>
>>     SMA.Transient.g_bytes 15.76K        -2.19K          .
>>     18.53K        15.50K        15.24K
>>
>>     SMF.s0.g_bytes 673.26M        47.95K          . 672.87M      
>>     669.89M       668.90M
>>
>>
>>     On 07/02/18 6:51 pm, Guillaume Quintard wrote:
>>>     Amount of storage used is just the sum of all the g_bytes fields
>>>
>>>     -- 
>>>     Guillaume Quintard
>>>
>>>     On Feb 7, 2018 14:12, "Pinakee BIswas" <pinakee at waltzz.com
>>>     <mailto:pinakee at waltzz.com>> wrote:
>>>
>>>         Hi Guillaume,
>>>
>>>         Thanks for your response and details.
>>>
>>>         Please find my comments inline:
>>>
>>>
>>>         On 07/02/18 6:21 pm, Guillaume Quintard wrote:
>>>>         Hi there,
>>>>
>>>>         So, varnish memory usage will mostly come from three
>>>>         directions:
>>>>         - storage, include Transient, so check the g_bytes fields
>>>>         in varnishstat. Passes and shortlived objects will use
>>>>         Transient, so you can either reduce those, or limit the
>>>>         Transient storage (unbounded by defaukt).
>>>         Checked the g_bytes for transient in varnishstat. It's in Kbs.
>>>>         - thread workspaces, one thread typically uses one
>>>>         workspace, so you can limit the number of threads, or
>>>>         reduce the workspace size
>>>>         - memory fragmentation: jemalloc will fragment up to 25%,
>>>>         not much you can do here.
>>>>
>>>>         And no, no way to modify the mmap without restarting varnish.
>>>>
>>>>         May I ask how much data you are caching?
>>>         Could you please let me know how to figure out the amount of
>>>         data cached?
>>>>
>>>>         -- 
>>>>         Guillaume Quintard
>>>>
>>>>         On Wed, Feb 7, 2018 at 1:34 PM, Pinakee BIswas
>>>>         <pinakee at waltzz.com <mailto:pinakee at waltzz.com>> wrote:
>>>>
>>>>             Hi,
>>>>
>>>>             We have been using varnish since more than a year for
>>>>             our ecommerce site.
>>>>
>>>>             Current version is 4.1.8.
>>>>
>>>>             OS is Linux. RAM is 8GB.
>>>>
>>>>             I am observing that the varnish memory usage is
>>>>             increasing (right now 7%) and so is resident memory
>>>>             usage increasing.
>>>>
>>>>             The storage being used is file:
>>>>
>>>>             -s file,/tmp/varnish/,${storage}
>>>>
>>>>             I am not sure about the reason for the increase in
>>>>             varnish memory usage:
>>>>
>>>>               * Is there a way to limit the varnish memory usage?
>>>>               * How can I diagnose what is consuming memory?
>>>>               * Most of our web pages have maximum 2 days of cache.
>>>>                 Also, some of the pages might be least visited.
>>>>               * Is there a way to manipulate varnish mmap (when
>>>>                 file storage is used)?
>>>>
>>>>             Would appreciate any help on the above for efficient
>>>>             use of varnish.
>>>>
>>>>             Thanks,
>>>>
>>>>             Pinakee
>>>>
>>>>
>>>>
>>>>             _______________________________________________
>>>>             varnish-misc mailing list
>>>>             varnish-misc at varnish-cache.org
>>>>             <mailto:varnish-misc at varnish-cache.org>
>>>>             https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
>>>>             <https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc>
>>>>
>>>>
>>>
>>
>

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/varnish-misc/attachments/20180208/94b5fbca/attachment.html>