varnish eats all RAM
reza at varnish-software.com
Wed May 5 13:10:10 UTC 2021
This is a known and unfortunate issue with the latest versions of jemalloc
and certain allocation patterns. You need to downgrade to jemalloc 3.6.
VP of Technology
On Wed, May 5, 2021 at 5:35 AM Marco Dickert - evolver group <
marco.dickert at evolver.de> wrote:
> Hi Geoff,
> thanks for your answer!
> On 2021-04-29 18:00:39, Geoff Simmons wrote:
> > You can also turn off THP system-wide (for all processes):
> > $ echo never > /sys/kernel/mm/transparent_hugepage/enabled
> > That might be your only option if the jemalloc version is too old.
> We diabled huge pages in the kernel, but this didn't solve the problem,
> the RAM
> consumption was unaffected.
> However, we found that the "transient storage" of varnish may be part of
> problem. At least, we could mitigate this behaviour by limiting the
> storage in the start parameters (last option):
> DAEMON_OPTS="-a :6081 \
> -T :6082 \
> -f /etc/varnish/default.vcl \
> -p ping_interval=6 -p cli_timeout=10 -p pipe_timeout=600 \
> -p listen_depth=4096 -p thread_pool_min=200 -p
> thread_pool_max=500 \
> -p workspace_client=128k -p nuke_limit=1000 \
> -S /etc/varnish/secret \
> -s malloc,6G \
> -s Transient=malloc,3G"
> Now varnish uses less RAM, and the varnishstats confirm that our limits
> SMA.s0.g_bytes 6.00G -107.61K . 6.00G 6.00G
> SMA.s0.g_space 132.70K 107.61K . 136.67K 137.35K
> SMA.Transient.g_bytes 1.55G 1022.19 . 1.55G 1.55G
> SMA.Transient.g_space 1.45G -1022.19 . 1.45G 1.45G
> However, varnish, in total, uses up to 12GB RAM instead of only 6GB
> (cache) +
> 3GB (transient). I tried to find a value in the varnishstat output which
> indicate how this additional RAM is used, but didn't find anything useful
> So two questions:
> 1) What might cause varnish to consume considerably more RAM than (cache +
> 2) What objects exactly are stored in the transient storage? The
> mentions "shortlived" objects  (the "shortlived" parameter is 200 in
> varnish, which seems to be the varnish debian package default, since we
> didn't set this explicitly), but I am not sure if that is limited to
> cacheable objects or or not. Also I don't know how to determine which
> requests lead to excessive usage of the transient storage.
> Thanks in advance for any further input.
> Mit freundlichen Grüßen
> Marco Dickert
> Administration und Technik
> evolver services GmbH
> Fon +49 / (0)3 71 / 4 00 03 78 24
> Fax +49 / (0)3 71 / 4 00 03 79
> E-Mail marco.dickert at evolver.de
> Web https://www.evolver.de
> Sitz der Gesellschaft: Chemnitz
> Handelsregister: Amtsgericht Chemnitz, HRB 22649
> Geschäftsführer: Torsten Gramann und Mathias Möckel
> varnish-misc mailing list
> varnish-misc at varnish-cache.org
-------------- next part --------------
An HTML attachment was scrubbed...
More information about the varnish-misc