Varnish suddenly started using much more memory

[Batanun B]

Hi,

About two weeks ago we deployed some minor changes to our Varnish servers in production, and after that we have noticed a big change in the memory that Varnish consumes.

Before the deploy, the amount of available memory on the servers were very stable, around 25 GB, for months on end. After the deploy, the amount of available memory dropped below 25 GB within 6 hours, and is dropping about 1 GB more each day, with no indication that it will level out before hitting rock bottom.

There was no change in traffic patterns during the time of the deploy. And we didn't change any OS or Varnish configuration. The deplow consisted only of trivial VCL changes, like changing the backend probe url to a dedicated healthcheck endpoint, and tweaking the ttl for a minor resource. Nothing of which could explain this massive change in memory usage.

We have configured varnish with "-s default=malloc,12G -s large=malloc,8G", where the combined 20GB is about 60% of the total server RAM of 32GB. This is below the recommended 75% maximum I've seen in many places.

Currently Varnish uses about 73% of the server memory, or 23GB (the RES column in htop). The default storage uses about 10 GB (SMA.default.g_bytes), while the large storage uses 8 GB. And the transient storage is currently about 2 MB (SMA.Transient.g_bytes). In total this results in about 18 GB. So what is that additional 5 GB used for? How can I troubleshoot that?

And, more importantly, what could possibly explain this sudden change?

The Ubuntu version stayed the same (20.04.5 LTS), and the Varnish version too (6.0.11-1~focal), as well as varnish-modules (0.15.1). I notice some differences in some installed packages of the servers, but nothing that stands out to me (but I'm no linux expert).

Regards
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/varnish-misc/attachments/20240516/d55ab61e/attachment.html>