varnish eats all RAM

[Marco Dickert - evolver group]

Hi Geoff,

thanks for your answer!

On 2021-04-29 18:00:39, Geoff Simmons wrote:
> You can also turn off THP system-wide (for all processes):
> $ echo never > /sys/kernel/mm/transparent_hugepage/enabled
> That might be your only option if the jemalloc version is too old.

We diabled huge pages in the kernel, but this didn't solve the problem, the RAM
consumption was unaffected.

However, we found that the "transient storage" of varnish may be part of our
problem. At least, we could mitigate this behaviour by limiting the transient
storage in the start parameters (last option):

-------
DAEMON_OPTS="-a :6081 \
             -T :6082 \
             -f /etc/varnish/default.vcl \
             -p ping_interval=6 -p cli_timeout=10 -p pipe_timeout=600 \
             -p listen_depth=4096 -p thread_pool_min=200 -p thread_pool_max=500 \
             -p workspace_client=128k -p nuke_limit=1000 \
             -S /etc/varnish/secret \
             -s malloc,6G \
             -s Transient=malloc,3G"
-------

Now varnish uses less RAM, and the varnishstats confirm that our limits should
work:

-------
SMA.s0.g_bytes            6.00G     -107.61K     .        6.00G     6.00G     6.00G
SMA.s0.g_space          132.70K      107.61K     .      136.67K   137.35K   137.35K
SMA.Transient.g_bytes     1.55G      1022.19     .        1.55G     1.55G     1.55G
SMA.Transient.g_space     1.45G     -1022.19     .        1.45G     1.45G     1.45G
-------

However, varnish, in total, uses up to 12GB RAM instead of only 6GB (cache) +
3GB (transient). I tried to find a value in the varnishstat output which might
indicate how this additional RAM is used, but didn't find anything useful yet.

So two questions:

1) What might cause varnish to consume considerably more RAM than (cache + transient
   storage)? 

2) What objects exactly are stored in the transient storage? The documentation
   mentions "shortlived" objects [1] (the "shortlived" parameter is 200 in our
   varnish, which seems to be the varnish debian package default, since we
   didn't set this explicitly), but I am not sure if that is limited to
   cacheable objects or or not. Also I don't know how to determine which
   requests lead to excessive usage of the transient storage.

Thanks in advance for any further input.

Cheers,
Marco

[1] https://varnish-cache.org/docs/trunk/users-guide/storage-backends.html#transient-storage

-- 
Mit freundlichen Grüßen
Marco Dickert

Administration und Technik
evolver services GmbH

Fon	+49 / (0)3 71 / 4 00 03 78 24
Fax	+49 / (0)3 71 / 4 00 03 79

E-Mail	marco.dickert at evolver.de
Web	https://www.evolver.de

Sitz der Gesellschaft:	Chemnitz
Handelsregister:	Amtsgericht Chemnitz, HRB 22649
Geschäftsführer:	Torsten Gramann und Mathias Möckel