<div dir="ltr"><div>Hi,</div><div><br></div><div>I feel like the answer is there, somewhere. You said that the deploy changed something, but that it can't possibly be the deploy.</div><div><br></div><div>I'm going to bet that it's the deploy. Most likely you changed something that messed up the willingness to cache, or your TTL.</div><div>First, check the difference in passes, if they are about the same, look for hit-for-misses, and lastly, look at how long Varnish is trying to cache the average object. I'm pretty one of those changed.</div><div><br></div><div>That being said, the memory shouldn't explode like that, which packages are you using?</div><div><br></div><div><div><div dir="ltr" class="gmail_signature" data-smartmail="gmail_signature"><div dir="ltr"><div>-- <br></div><div>Guillaume Quintard<br></div></div></div></div><br></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Thu, May 16, 2024 at 2:19 AM Batanun B <<a href="mailto:batanun@hotmail.com">batanun@hotmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div class="msg4664428758100320052">
<div dir="ltr">
<div style="font-family:Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
Hi,</div>
<div style="font-family:Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
About two weeks ago we deployed some minor changes to our Varnish servers in production, and after that we have noticed a big change in the memory that Varnish consumes.</div>
<div style="font-family:Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
Before the deploy, the amount of available memory on the servers were very stable, around 25 GB, for months on end. After the deploy, the amount of available memory dropped below 25 GB within 6 hours, and is dropping about 1 GB more each day, with no indication
that it will level out before hitting rock bottom.</div>
<div style="font-family:Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
There was no change in traffic patterns during the time of the deploy. And we didn't change any OS or Varnish configuration. The deplow consisted only of trivial VCL changes, like changing the backend probe url to a dedicated healthcheck endpoint, and tweaking
the ttl for a minor resource. Nothing of which could explain this massive change in memory usage.</div>
<div style="font-family:Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
We have configured varnish with "-s default=malloc,12G -s large=malloc,8G", where the combined 20GB is about 60% of the total server RAM of 32GB. This is below the recommended 75% maximum I've seen in many places.</div>
<div style="font-family:Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
Currently Varnish uses about 73% of the server memory, or 23GB (the RES column in htop). The default storage uses about 10 GB (SMA.default.g_bytes), while the large storage uses 8 GB. And the transient storage is currently about 2 MB (SMA.Transient.g_bytes).
In total this results in about 18 GB. So what is that additional 5 GB used for? How can I troubleshoot that?</div>
<div style="font-family:Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
And, more importantly, what could possibly explain this sudden change?</div>
<div style="font-family:Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
The Ubuntu version stayed the same (20.04.5 LTS), and the Varnish version too (6.0.11-1~focal), as well as varnish-modules (0.15.1). I notice some differences in some installed packages of the servers, but nothing that stands out to me (but I'm no linux expert).</div>
<div style="font-family:Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
Regards</div>
</div>
_______________________________________________<br>
varnish-misc mailing list<br>
<a href="mailto:varnish-misc@varnish-cache.org" target="_blank">varnish-misc@varnish-cache.org</a><br>
<a href="https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc" rel="noreferrer" target="_blank">https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc</a><br>
</div></blockquote></div>