Varnish Cache Runaway RAM Usage!

Tue Sep 7 15:43:17 UTC 2021

On Tue, Sep 7, 2021 at 3:14 PM John Kormanec <jpkorma at yahoo.com> wrote:
>
> Hello,
>
> We have been using the open source (free) version of Varnish Cache for my company's website for several months but are repeatedly running into an issue where Varnish's memory usage increases until all of the server's available memory is consumed, rendering the server unresponsive.ï¿½ This continues to happen despite having Varnish's malloc and transient malloc storage settings dialed way down (currently set at 50% combined of available RAM).ï¿½ Here's an excerpt from our storage backend configuration showing these settings.ï¿½ I've attached our full backend storage settings to this message for review.
>
> # Configure Varnish listening port, default "vcl" file location, memory allocation, etc:
> ExecStart=/usr/sbin/varnishd -a :80 -f /etc/varnish/default.vcl -s malloc,14G -s Transient=malloc,512M -T 0.0.0.0:2000
> ExecReload=/usr/sbin/varnishreload
>
> We're running version 6.0.2 of Varnish Cache on a CentOS 8 virtual server.ï¿½ï¿½ The server has 30 GB of RAM, 29.3 GB of which are available for use.

You should upgrade to 6.0.8 to get the latest bug fixes for the 6.0 branch.

> I've read a number of technical posts from other Varnish users complaining about the same issue, including this one:ï¿½ï¿½https://stackoverflow.com/questions/65450044/varnish-6-lts-w-centos-8-not-respecting-memory-limits.ï¿½ Unfortunately, I've not seen any published resolution for this problem, but several tuning articles I've read talk about adjusting the server's memory allocation manager to more aggressively cleanup fragmentation, expired objects, etc.ï¿½ï¿½ Here's one such post that talks about adjusting the "jemalloc" memory allocation manager settings as a possible fix:ï¿½ https://info.varnish-software.com/blog/understanding-varnish-cache-memory-usage.ï¿½ I searched our CentOS 8 server to see what packages are installed but jemalloc is not in the list.

I think you would have better results with jemalloc 3.6 that better
fits Varnish workloads, but EPEL8 ships 5.2.1 currently.

One thing to consider besides the jemalloc overhead is that you are
only limiting cache storage to 14.5GB and there will be a memory
footprint for everything else.

> I'm still relatively new to Varnish Cache and am not sure what the next steps should be for troubleshooting & identifying the issue.ï¿½ FWIW, I reached out to the folks at Varnish Cache asking if they could offer any suggestions, but they said we'd have to upgrade to their Enterprise version, which uses something called a "massive storage engine" that would eliminate this problem.ï¿½ Not sure what the differences are between the paid / free versions, but I'm hoping to find a solution to this problem here before having to upgrade.
>
> Thanks in advance for any assistance the community can provide.

Full disclosure, I work for Varnish Software, but what you were told
was correct.

The Massive Storage Engine brings a feature called Memory Governor
that will allow your Varnish instance to pull all the levers available
to decrease its footprint back to 15GB whenever it crosses the
threshold. You could probably also expect less jemalloc overhead out
of the box and have a decent memory target for 15GB of storage, for
example 20GB.

With Varnish Cache alone it is more difficult to plan for memory usage.

Dridi