varnish eats all RAM

[Reza Naghibi]

This is a known and unfortunate issue with the latest versions of jemalloc
and certain allocation patterns. You need to downgrade to jemalloc 3.6.

https://github.com/varnishcache/varnish-cache/issues/3511#issuecomment-771592238

--
Reza Naghibi
VP of Technology
Varnish Software


On Wed, May 5, 2021 at 5:35 AM Marco Dickert - evolver group <
marco.dickert at evolver.de> wrote:

> Hi Geoff,
>
> thanks for your answer!
>
> On 2021-04-29 18:00:39, Geoff Simmons wrote:
> > You can also turn off THP system-wide (for all processes):
> > $ echo never > /sys/kernel/mm/transparent_hugepage/enabled
> > That might be your only option if the jemalloc version is too old.
>
> We diabled huge pages in the kernel, but this didn't solve the problem,
> the RAM
> consumption was unaffected.
>
> However, we found that the "transient storage" of varnish may be part of
> our
> problem. At least, we could mitigate this behaviour by limiting the
> transient
> storage in the start parameters (last option):
>
> -------
> DAEMON_OPTS="-a :6081 \
>              -T :6082 \
>              -f /etc/varnish/default.vcl \
>              -p ping_interval=6 -p cli_timeout=10 -p pipe_timeout=600 \
>              -p listen_depth=4096 -p thread_pool_min=200 -p
> thread_pool_max=500 \
>              -p workspace_client=128k -p nuke_limit=1000 \
>              -S /etc/varnish/secret \
>              -s malloc,6G \
>              -s Transient=malloc,3G"
> -------
>
> Now varnish uses less RAM, and the varnishstats confirm that our limits
> should
> work:
>
> -------
> SMA.s0.g_bytes            6.00G     -107.61K     .        6.00G     6.00G
>    6.00G
> SMA.s0.g_space          132.70K      107.61K     .      136.67K   137.35K
>  137.35K
> SMA.Transient.g_bytes     1.55G      1022.19     .        1.55G     1.55G
>    1.55G
> SMA.Transient.g_space     1.45G     -1022.19     .        1.45G     1.45G
>    1.45G
> -------
>
> However, varnish, in total, uses up to 12GB RAM instead of only 6GB
> (cache) +
> 3GB (transient). I tried to find a value in the varnishstat output which
> might
> indicate how this additional RAM is used, but didn't find anything useful
> yet.
>
> So two questions:
>
> 1) What might cause varnish to consume considerably more RAM than (cache +
> transient
>    storage)?
>
> 2) What objects exactly are stored in the transient storage? The
> documentation
>    mentions "shortlived" objects [1] (the "shortlived" parameter is 200 in
> our
>    varnish, which seems to be the varnish debian package default, since we
>    didn't set this explicitly), but I am not sure if that is limited to
>    cacheable objects or or not. Also I don't know how to determine which
>    requests lead to excessive usage of the transient storage.
>
> Thanks in advance for any further input.
>
> Cheers,
> Marco
>
> [1]
> https://varnish-cache.org/docs/trunk/users-guide/storage-backends.html#transient-storage
>
> --
> Mit freundlichen Grüßen
> Marco Dickert
>
> Administration und Technik
> evolver services GmbH
>
> Fon     +49 / (0)3 71 / 4 00 03 78 24
> Fax     +49 / (0)3 71 / 4 00 03 79
>
> E-Mail  marco.dickert at evolver.de
> Web     https://www.evolver.de
>
> Sitz der Gesellschaft:  Chemnitz
> Handelsregister:        Amtsgericht Chemnitz, HRB 22649
> Geschäftsführer:        Torsten Gramann und Mathias Möckel
> _______________________________________________
> varnish-misc mailing list
> varnish-misc at varnish-cache.org
> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/varnish-misc/attachments/20210505/92a4dc7a/attachment.html>