Varnish suddenly started using much more memory

[Guillaume Quintard]

Hi,

I feel like the answer is there, somewhere. You said that the deploy
changed something, but that it can't possibly be the deploy.

I'm going to bet that it's the deploy. Most likely you changed something
that messed up the willingness to cache, or your TTL.
First, check the difference in passes, if they are about the same, look for
hit-for-misses, and lastly, look at how long Varnish is trying to cache the
average object. I'm pretty one of those changed.

That being said, the memory shouldn't explode like that, which packages are
you using?

-- 
Guillaume Quintard


On Thu, May 16, 2024 at 2:19 AM Batanun B <batanun at hotmail.com> wrote:

> Hi,
>
> About two weeks ago we deployed some minor changes to our Varnish servers
> in production, and after that we have noticed a big change in the memory
> that Varnish consumes.
>
> Before the deploy, the amount of available memory on the servers were very
> stable, around 25 GB, for months on end. After the deploy, the amount of
> available memory dropped below 25 GB within 6 hours, and is dropping about
> 1 GB more each day, with no indication that it will level out before
> hitting rock bottom.
>
> There was no change in traffic patterns during the time of the deploy. And
> we didn't change any OS or Varnish configuration. The deplow consisted only
> of trivial VCL changes, like changing the backend probe url to a dedicated
> healthcheck endpoint, and tweaking the ttl for a minor resource. Nothing of
> which could explain this massive change in memory usage.
>
> We have configured varnish with "-s default=malloc,12G -s
> large=malloc,8G", where the combined 20GB is about 60% of the total server
> RAM of 32GB. This is below the recommended 75% maximum I've seen in many
> places.
>
> Currently Varnish uses about 73% of the server memory, or 23GB (the RES
> column in htop). The default storage uses about 10 GB
> (SMA.default.g_bytes), while the large storage uses 8 GB. And the transient
> storage is currently about 2 MB (SMA.Transient.g_bytes). In total this
> results in about 18 GB. So what is that additional 5 GB used for? How can I
> troubleshoot that?
>
> And, more importantly, what could possibly explain this sudden change?
>
> The Ubuntu version stayed the same (20.04.5 LTS), and the Varnish version
> too (6.0.11-1~focal), as well as varnish-modules (0.15.1). I notice some
> differences in some installed packages of the servers, but nothing that
> stands out to me (but I'm no linux expert).
>
> Regards
> _______________________________________________
> varnish-misc mailing list
> varnish-misc at varnish-cache.org
> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/varnish-misc/attachments/20240516/65c29f04/attachment.html>