<div dir="ltr"><div>Sorry Batanun, this thread got lost in my inbox. Would you be able to upgrade to 7.5 and see if you get the same results? I'm pretty sure it's a jemalloc issue, but upgrading should make it clear.</div><div>You are on Ubuntu, right? Which version?<br></div><div><div><div dir="ltr" class="gmail_signature" data-smartmail="gmail_signature"><div dir="ltr"><div>-- <br></div><div>Guillaume Quintard<br></div></div></div></div><br></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Mon, May 20, 2024 at 1:50 AM Batanun B <<a href="mailto:batanun@hotmail.com">batanun@hotmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">> Sorry, I should have been clearer, I meant: where are the varnish packages coming from? Are they from the official repositories, from <a href="https://packagecloud.io/varnishcache/" rel="noreferrer" target="_blank">https://packagecloud.io/varnishcache/</a> or built from source maybe?<br>
<br>
Ah, I see. They come from varnishcache packagecloud. More specifically, we use:<br>
<br>
<a href="https://packagecloud.io/install/repositories/varnishcache/varnish60lts/script.deb.sh" rel="noreferrer" target="_blank">https://packagecloud.io/install/repositories/varnishcache/varnish60lts/script.deb.sh</a><br>
<br>
<br>
> you should really invest some time in something like prometheus, it would probably have made the issue obvious<br>
<br>
Yes, in hindsight we definitely should have done that. I will discuss this with my coworkers going forward.<br>
<br>
<br>
> Is there any chance you can run the old version on the server to explore the differences?<br>
<br>
Possibly, for a limited time. If so, what types of tests would I do? And how long time would I need to run the old version?<br>
<br>
Note that with our setup, we wouldn't be able to run two different images at the same time, in the same environment, with both recieving traffic. So all traffic would be routed to this version (multiple servers, but all running the same image).<br>
<br>
An alternative approach that I'm considering, is to switch to the old image, but manually update the VCL to the new version. If the problem remains, then the issue is almost certainly with the VLC. But if the problem disapears, then it's more likely something else.<br>
<br>
<br>
> what's the output of: varnishstat -1 -f '*g_bytes'<br>
<br>
SMA.default.g_bytes 10951750929 . Bytes outstanding<br>
SMA.large.g_bytes 8587329728 . Bytes outstanding<br>
SMA.Transient.g_bytes 3177920 . Bytes outstanding<br>
<br>
So, the default storage usage has gone up with 2GB since my first message here, while the others have remained the same. Meanwhile, the total memory usage of Varnish has gone up to 26 GB, an increase of 3 GB. So now the overhead has gone up with 1GB to a total of 6 GB.<br>
<br>
Going forward, it will be interesting to see how the memory consumption changes after the default storage has reached its max (2 GB from where it is now). If we're lucky, it will stabilize, and then I'm not sure if it's worth it to troubleshoot any further. Otherwise, the free memory would get a bit too close to zero for our comfort, with no indication of stopping. <br>
<br>
Does Varnish keep track of total available OS memory, and start releasing memory by throwing out objects from the cache? Or will it continue to eat memory until something fails?<br>
<br>
<br>
> have you tweaked any workspaces/thread parameters?<br>
<br>
Nope. As I said, we haven't changed any OS or Varnish configuration.<br>
_______________________________________________<br>
varnish-misc mailing list<br>
<a href="mailto:varnish-misc@varnish-cache.org" target="_blank">varnish-misc@varnish-cache.org</a><br>
<a href="https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc" rel="noreferrer" target="_blank">https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc</a><br>
</blockquote></div>