Varnish suddenly started using much more memory

[Batanun B]

> Sorry, I should have been clearer, I meant: where are the varnish packages coming from? Are they from the official repositories, from https://packagecloud.io/varnishcache/ or built from source maybe?

Ah, I see. They come from varnishcache packagecloud. More specifically, we use:

https://packagecloud.io/install/repositories/varnishcache/varnish60lts/script.deb.sh


> you should really invest some time in something like prometheus, it would probably have made the issue obvious

Yes, in hindsight we definitely should have done that. I will discuss this with my coworkers going forward.


> Is there any chance you can run the old version on the server to explore the differences?

Possibly, for a limited time. If so, what types of tests would I do? And how long time would I need to run the old version?

Note that with our setup, we wouldn't be able to run two different images at the same time, in the same environment, with both recieving traffic. So all traffic would be routed to this version (multiple servers, but all running the same image).

An alternative approach that I'm considering, is to switch to the old image, but manually update the VCL to the new version. If the problem remains, then the issue is almost certainly with the VLC. But if the problem disapears, then it's more likely something else.


> what's the output of: varnishstat -1 -f '*g_bytes'

SMA.default.g_bytes  10951750929          .   Bytes outstanding
SMA.large.g_bytes     8587329728          .   Bytes outstanding
SMA.Transient.g_bytes      3177920          .   Bytes outstanding

So, the default storage usage has gone up with 2GB since my first message here, while the others have remained the same. Meanwhile, the total memory usage of Varnish has gone up to 26 GB, an increase of 3 GB. So now the overhead has gone up with 1GB to a total of 6 GB.

Going forward, it will be interesting to see how the memory consumption changes after the default storage has reached its max (2 GB from where it is now). If we're lucky, it will stabilize, and then I'm not sure if it's worth it to troubleshoot any further. Otherwise, the free memory would get a bit too close to zero for our comfort, with no indication of stopping. 

Does Varnish keep track of total available OS memory, and start releasing memory by throwing out objects from the cache? Or will it continue to eat memory until something fails?


> have you tweaked any workspaces/thread parameters?

Nope. As I said, we haven't changed any OS or Varnish configuration.