Varnish suddenly started using much more memory

[Guillaume Quintard]

Sorry Batanun, this thread got lost in my inbox. Would you be able to
upgrade to 7.5 and see if you get the same results? I'm pretty sure it's a
jemalloc issue, but upgrading should make it clear.
You are on Ubuntu, right? Which version?
-- 
Guillaume Quintard


On Mon, May 20, 2024 at 1:50 AM Batanun B <batanun at hotmail.com> wrote:

> > Sorry, I should have been clearer, I meant: where are the varnish
> packages coming from? Are they from the official repositories, from
> https://packagecloud.io/varnishcache/ or built from source maybe?
>
> Ah, I see. They come from varnishcache packagecloud. More specifically, we
> use:
>
>
> https://packagecloud.io/install/repositories/varnishcache/varnish60lts/script.deb.sh
>
>
> > you should really invest some time in something like prometheus, it
> would probably have made the issue obvious
>
> Yes, in hindsight we definitely should have done that. I will discuss this
> with my coworkers going forward.
>
>
> > Is there any chance you can run the old version on the server to explore
> the differences?
>
> Possibly, for a limited time. If so, what types of tests would I do? And
> how long time would I need to run the old version?
>
> Note that with our setup, we wouldn't be able to run two different images
> at the same time, in the same environment, with both recieving traffic. So
> all traffic would be routed to this version (multiple servers, but all
> running the same image).
>
> An alternative approach that I'm considering, is to switch to the old
> image, but manually update the VCL to the new version. If the problem
> remains, then the issue is almost certainly with the VLC. But if the
> problem disapears, then it's more likely something else.
>
>
> > what's the output of: varnishstat -1 -f '*g_bytes'
>
> SMA.default.g_bytes  10951750929          .   Bytes outstanding
> SMA.large.g_bytes     8587329728          .   Bytes outstanding
> SMA.Transient.g_bytes      3177920          .   Bytes outstanding
>
> So, the default storage usage has gone up with 2GB since my first message
> here, while the others have remained the same. Meanwhile, the total memory
> usage of Varnish has gone up to 26 GB, an increase of 3 GB. So now the
> overhead has gone up with 1GB to a total of 6 GB.
>
> Going forward, it will be interesting to see how the memory consumption
> changes after the default storage has reached its max (2 GB from where it
> is now). If we're lucky, it will stabilize, and then I'm not sure if it's
> worth it to troubleshoot any further. Otherwise, the free memory would get
> a bit too close to zero for our comfort, with no indication of stopping.
>
> Does Varnish keep track of total available OS memory, and start releasing
> memory by throwing out objects from the cache? Or will it continue to eat
> memory until something fails?
>
>
> > have you tweaked any workspaces/thread parameters?
>
> Nope. As I said, we haven't changed any OS or Varnish configuration.
> _______________________________________________
> varnish-misc mailing list
> varnish-misc at varnish-cache.org
> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/varnish-misc/attachments/20240613/b4f379fd/attachment.html>