Memory utilisation gradually increasing

[FULLER, David]

Thanks for your reply.

Am I correct in thinking that even though we’re not using Varnish for caching, objects are still stored using malloc?  We have malloc set at 100MB, with 1GB allocated to the Varnish container.  Based on the docs you’ve linked to should we set malloc at 750MB (75% of the container memory) and could this be the cause of the memory problem we’re seeing?

Regards,
David

From: Guillaume Quintard <guillaume at varnish-software.com>
Date: Wednesday, 25 July 2018 at 18:30
To: "FULLER, David" <FULLERD at parliament.uk>, varnish-misc <varnish-misc at varnish-cache.org>
Subject: Re: Memory utilisation gradually increasing

Let's keep the mailing list in CC :-)

http://varnish-cache.org/docs/trunk/users-guide/storage-backends.html#transient-storage<http://varnish-cache.org/docs/trunk/users-guide/storage-backends.html#transient-storage>

You also have Reza's post: https://info.varnish-software.com/blog/understanding-varnish-cache-memory-usage<https://info.varnish-software.com/blog/understanding-varnish-cache-memory-usage>

Finally, memory is will also be consumed by workspaces (one per thread).

--
Guillaume Quintard

On Wed, Jul 25, 2018 at 8:32 AM, FULLER, David <FULLERD at parliament.uk<mailto:FULLERD at parliament.uk>> wrote:
Hi Guillaume,

Thanks for the response, I’ve run the command you’ve suggested and get the following:

/ # varnishstat -1 | grep -e g_space -e g_bytes
SMA.s0.g_bytes                                0          .   Bytes outstanding
SMA.s0.g_space                        104857600          .   Bytes available
SMA.Transient.g_bytes                         0          .   Bytes outstanding
SMA.Transient.g_space                         0          .   Bytes available

The Varnish container was redeployed this afternoon and currently shows memory utilisation around 3% so probably not illustrating the problem very well right now.

Is there a way to limit the amount of transient storage and clear when hit without effecting performance?  Given that we aren’t caching are there any other settings we should look at to improve memory utilisation?

Kind regards,
David



From: Guillaume Quintard <guillaume at varnish-software.com<mailto:guillaume at varnish-software.com>>
Date: Wednesday, 25 July 2018 at 16:00
To: "FULLER, David" <FULLERD at parliament.uk<mailto:FULLERD at parliament.uk>>
Cc: "varnish-misc at varnish-cache.org<mailto:varnish-misc at varnish-cache.org>" <varnish-misc at varnish-cache.org<mailto:varnish-misc at varnish-cache.org>>
Subject: Re: Memory utilisation gradually increasing

Hello David,

Have a look at varnishstat ("varnishstat -1 | grep -e g_space -e g_bytes"). When you are passing, varnish is going to consume Transient storage.

--
Guillaume Quintard

On Wed, Jul 25, 2018 at 7:19 AM, FULLER, David <FULLERD at parliament.uk<mailto:FULLERD at parliament.uk>> wrote:
We currently have an issue with memory utilisation in Varnish 5.2.1, we are only using Reverse Proxy not the caching functionality.

We are running it in an AWS ECS Docker container, with 1GB of memory allocated.  Memory increases daily by around 8% until it tops out and site connectivity problems occur.  Redeploying the container resolves the problem and the cycle starts again.

When Varnish starts we have ‘malloc’ set at 100MB, from my understanding this setting is only relevant if caching is being used, which in our case it isn’t.

Has anyone seen a similar problem?

Thanks

UK Parliament Disclaimer: This e-mail is confidential to the intended recipient. If you have received it in error, please notify the sender and delete it from your system. Any unauthorised use, disclosure, or copying is not permitted. This e-mail has been checked for viruses, but no liability is accepted for any damage caused by any virus transmitted by this e-mail. This e-mail address is not secure, is not encrypted and should not be used for sensitive data.

_______________________________________________
varnish-misc mailing list
varnish-misc at varnish-cache.org<mailto:varnish-misc at varnish-cache.org>
https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc<https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc>

UK Parliament Disclaimer: This e-mail is confidential to the intended recipient. If you have received it in error, please notify the sender and delete it from your system. Any unauthorised use, disclosure, or copying is not permitted. This e-mail has been checked for viruses, but no liability is accepted for any damage caused by any virus transmitted by this e-mail. This e-mail address is not secure, is not encrypted and should not be used for sensitive data.

UK Parliament Disclaimer: This e-mail is confidential to the intended recipient. If you have received it in error, please notify the sender and delete it from your system. Any unauthorised use, disclosure, or copying is not permitted. This e-mail has been checked for viruses, but no liability is accepted for any damage caused by any virus transmitted by this e-mail. This e-mail address is not secure, is not encrypted and should not be used for sensitive data.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/varnish-misc/attachments/20180726/43fc504c/attachment-0001.html>