<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1">
<style type="text/css" style="display:none;"> P {margin-top:0;margin-bottom:0;} </style>
</head>
<body dir="ltr">
<div class="elementToProof" style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
Hi,</div>
<div style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
About two weeks ago we deployed some minor changes to our Varnish servers in production, and after that we have noticed a big change in the memory that Varnish consumes.</div>
<div style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
Before the deploy, the amount of available memory on the servers were very stable, around 25 GB, for months on end. After the deploy, the amount of available memory dropped below 25 GB within 6 hours, and is dropping about 1 GB more each day, with no indication
that it will level out before hitting rock bottom.</div>
<div style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
There was no change in traffic patterns during the time of the deploy. And we didn't change any OS or Varnish configuration. The deplow consisted only of trivial VCL changes, like changing the backend probe url to a dedicated healthcheck endpoint, and tweaking
the ttl for a minor resource. Nothing of which could explain this massive change in memory usage.</div>
<div style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div class="elementToProof" style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
We have configured varnish with "-s default=malloc,12G -s large=malloc,8G", where the combined 20GB is about 60% of the total server RAM of 32GB. This is below the recommended 75% maximum I've seen in many places.</div>
<div style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div class="elementToProof" style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
Currently Varnish uses about 73% of the server memory, or 23GB (the RES column in htop). The default storage uses about 10 GB (SMA.default.g_bytes), while the large storage uses 8 GB. And the transient storage is currently about 2 MB (SMA.Transient.g_bytes).
In total this results in about 18 GB. So what is that additional 5 GB used for? How can I troubleshoot that?</div>
<div style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
And, more importantly, what could possibly explain this sudden change?</div>
<div style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
The Ubuntu version stayed the same (20.04.5 LTS), and the Varnish version too (6.0.11-1~focal), as well as varnish-modules (0.15.1). I notice some differences in some installed packages of the servers, but nothing that stands out to me (but I'm no linux expert).</div>
<div style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div class="elementToProof" style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
Regards</div>
</body>
</html>