<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<style type="text/css" style="display:none;"> P {margin-top:0;margin-bottom:0;} </style>
</head>
<body dir="ltr">
<div class="elementToProof" style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
Hi Guillaume,</div>
<div class="elementToProof" style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div class="elementToProof" style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
This time it is I who needs to apologize for a late reply (7 months later).</div>
<div class="elementToProof" style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div class="elementToProof" style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
We are strictly on the LTS diet, I'm afraid. So we can't upgrade the major version until there is a new LTS series of Varnish. So it's 6.0.x for a while still I guess. We are on Ubuntu 20 though, and I guess we could update to the latest LTS version there.</div>
<div class="elementToProof" style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div id="appendonsend"></div>
<div class="elementToProof" style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<hr style="display: inline-block; width: 98%;">
<div id="divRplyFwdMsg" dir="ltr"><span style="font-family: Calibri, sans-serif; font-size: 11pt; color: rgb(0, 0, 0);"><b>From:</b> Guillaume Quintard <guillaume.quintard@gmail.com><br>
<b>Sent:</b> Thursday, June 13, 2024 10:07 PM<br>
<b>To:</b> Batanun B <batanun@hotmail.com><br>
<b>Cc:</b> varnish-misc@varnish-cache.org <varnish-misc@varnish-cache.org><br>
<b>Subject:</b> Re: Varnish suddenly started using much more memory</span>
<div> </div>
</div>
<div style="direction: ltr;">Sorry Batanun, this thread got lost in my inbox. Would you be able to upgrade to 7.5 and see if you get the same results? I'm pretty sure it's a jemalloc issue, but upgrading should make it clear.</div>
<div style="direction: ltr;">You are on Ubuntu, right? Which version?</div>
<div style="direction: ltr;">--</div>
<div class="elementToProof" style="direction: ltr;">Guillaume Quintard</div>
<div style="direction: ltr;"><br>
</div>
<br>
<div style="direction: ltr;">On Mon, May 20, 2024 at 1:50 AM Batanun B <<a href="mailto:batanun@hotmail.com" id="OWA3171926e-9c9d-8ce6-45ab-e3362f561ef6" class="OWAAutoLink">batanun@hotmail.com</a>> wrote:</div>
<blockquote style="margin: 0px 0px 0px 0.8ex; padding-left: 1ex; border-left: 1px solid rgb(204, 204, 204);">
> Sorry, I should have been clearer, I meant: where are the varnish packages coming from? Are they from the official repositories, from
<a href="https://packagecloud.io/varnishcache/" id="OWA73e93e6d-738a-0a5e-5bcd-c79c2ca7af33" class="OWAAutoLink" data-auth="NotApplicable">
https://packagecloud.io/varnishcache/</a> or built from source maybe?<br>
<br>
Ah, I see. They come from varnishcache packagecloud. More specifically, we use:<br>
<br>
<a href="https://packagecloud.io/install/repositories/varnishcache/varnish60lts/script.deb.sh" id="OWA23b296be-3aa1-7201-b984-f0e43d744830" class="OWAAutoLink" data-auth="NotApplicable">https://packagecloud.io/install/repositories/varnishcache/varnish60lts/script.deb.sh</a><br>
<br>
<br>
> you should really invest some time in something like prometheus, it would probably have made the issue obvious<br>
<br>
Yes, in hindsight we definitely should have done that. I will discuss this with my coworkers going forward.<br>
<br>
<br>
> Is there any chance you can run the old version on the server to explore the differences?<br>
<br>
Possibly, for a limited time. If so, what types of tests would I do? And how long time would I need to run the old version?<br>
<br>
Note that with our setup, we wouldn't be able to run two different images at the same time, in the same environment, with both recieving traffic. So all traffic would be routed to this version (multiple servers, but all running the same image).<br>
<br>
An alternative approach that I'm considering, is to switch to the old image, but manually update the VCL to the new version. If the problem remains, then the issue is almost certainly with the VLC. But if the problem disapears, then it's more likely something
else.<br>
<br>
<br>
> what's the output of: varnishstat -1 -f '*g_bytes'<br>
<br>
SMA.default.g_bytes 10951750929 . Bytes outstanding<br>
SMA.large.g_bytes 8587329728 . Bytes outstanding<br>
SMA.Transient.g_bytes 3177920 . Bytes outstanding<br>
<br>
So, the default storage usage has gone up with 2GB since my first message here, while the others have remained the same. Meanwhile, the total memory usage of Varnish has gone up to 26 GB, an increase of 3 GB. So now the overhead has gone up with 1GB to a total
of 6 GB.<br>
<br>
Going forward, it will be interesting to see how the memory consumption changes after the default storage has reached its max (2 GB from where it is now). If we're lucky, it will stabilize, and then I'm not sure if it's worth it to troubleshoot any further.
Otherwise, the free memory would get a bit too close to zero for our comfort, with no indication of stopping.<br>
<br>
Does Varnish keep track of total available OS memory, and start releasing memory by throwing out objects from the cache? Or will it continue to eat memory until something fails?<br>
<br>
<br>
> have you tweaked any workspaces/thread parameters?<br>
<br>
Nope. As I said, we haven't changed any OS or Varnish configuration.<br>
_______________________________________________<br>
varnish-misc mailing list<br>
<a href="mailto:varnish-misc@varnish-cache.org" id="OWAf5cf5dbf-b34d-794b-3d37-bb7dfacf1042" class="OWAAutoLink">varnish-misc@varnish-cache.org</a><br>
<a href="https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc" id="OWA3da7d4b1-6b60-1335-0bc8-18c14bf87b07" class="OWAAutoLink" data-auth="NotApplicable">https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc</a><br>
</blockquote>
</body>
</html>