From marco.dickert at evolver.de Wed May 5 09:36:38 2021 From: marco.dickert at evolver.de (Marco Dickert - evolver group) Date: Wed, 5 May 2021 11:36:38 +0200 Subject: varnish eats all RAM In-Reply-To: <0985b590-037b-eed5-0500-1b216bed6451@uplex.de> References: <20210429152103.GD3147@marco.evolver.de> <0985b590-037b-eed5-0500-1b216bed6451@uplex.de> Message-ID: <20210505093638.GA4667@marco.evolver.de> Hi Geoff, thanks for your answer! On 2021-04-29 18:00:39, Geoff Simmons wrote: > You can also turn off THP system-wide (for all processes): > $ echo never > /sys/kernel/mm/transparent_hugepage/enabled > That might be your only option if the jemalloc version is too old. We diabled huge pages in the kernel, but this didn't solve the problem, the RAM consumption was unaffected. However, we found that the "transient storage" of varnish may be part of our problem. At least, we could mitigate this behaviour by limiting the transient storage in the start parameters (last option): ------- DAEMON_OPTS="-a :6081 \ -T :6082 \ -f /etc/varnish/default.vcl \ -p ping_interval=6 -p cli_timeout=10 -p pipe_timeout=600 \ -p listen_depth=4096 -p thread_pool_min=200 -p thread_pool_max=500 \ -p workspace_client=128k -p nuke_limit=1000 \ -S /etc/varnish/secret \ -s malloc,6G \ -s Transient=malloc,3G" ------- Now varnish uses less RAM, and the varnishstats confirm that our limits should work: ------- SMA.s0.g_bytes 6.00G -107.61K . 6.00G 6.00G 6.00G SMA.s0.g_space 132.70K 107.61K . 136.67K 137.35K 137.35K SMA.Transient.g_bytes 1.55G 1022.19 . 1.55G 1.55G 1.55G SMA.Transient.g_space 1.45G -1022.19 . 1.45G 1.45G 1.45G ------- However, varnish, in total, uses up to 12GB RAM instead of only 6GB (cache) + 3GB (transient). I tried to find a value in the varnishstat output which might indicate how this additional RAM is used, but didn't find anything useful yet. So two questions: 1) What might cause varnish to consume considerably more RAM than (cache + transient storage)? 2) What objects exactly are stored in the transient storage? The documentation mentions "shortlived" objects [1] (the "shortlived" parameter is 200 in our varnish, which seems to be the varnish debian package default, since we didn't set this explicitly), but I am not sure if that is limited to cacheable objects or or not. Also I don't know how to determine which requests lead to excessive usage of the transient storage. Thanks in advance for any further input. Cheers, Marco [1] https://varnish-cache.org/docs/trunk/users-guide/storage-backends.html#transient-storage -- Mit freundlichen Gr??en Marco Dickert Administration und Technik evolver services GmbH Fon +49 / (0)3 71 / 4 00 03 78 24 Fax +49 / (0)3 71 / 4 00 03 79 E-Mail marco.dickert at evolver.de Web https://www.evolver.de Sitz der Gesellschaft: Chemnitz Handelsregister: Amtsgericht Chemnitz, HRB 22649 Gesch?ftsf?hrer: Torsten Gramann und Mathias M?ckel From reza at varnish-software.com Wed May 5 13:10:10 2021 From: reza at varnish-software.com (Reza Naghibi) Date: Wed, 5 May 2021 09:10:10 -0400 Subject: varnish eats all RAM In-Reply-To: <20210505093638.GA4667@marco.evolver.de> References: <20210429152103.GD3147@marco.evolver.de> <0985b590-037b-eed5-0500-1b216bed6451@uplex.de> <20210505093638.GA4667@marco.evolver.de> Message-ID: This is a known and unfortunate issue with the latest versions of jemalloc and certain allocation patterns. You need to downgrade to jemalloc 3.6. https://github.com/varnishcache/varnish-cache/issues/3511#issuecomment-771592238 -- Reza Naghibi VP of Technology Varnish Software On Wed, May 5, 2021 at 5:35 AM Marco Dickert - evolver group < marco.dickert at evolver.de> wrote: > Hi Geoff, > > thanks for your answer! > > On 2021-04-29 18:00:39, Geoff Simmons wrote: > > You can also turn off THP system-wide (for all processes): > > $ echo never > /sys/kernel/mm/transparent_hugepage/enabled > > That might be your only option if the jemalloc version is too old. > > We diabled huge pages in the kernel, but this didn't solve the problem, > the RAM > consumption was unaffected. > > However, we found that the "transient storage" of varnish may be part of > our > problem. At least, we could mitigate this behaviour by limiting the > transient > storage in the start parameters (last option): > > ------- > DAEMON_OPTS="-a :6081 \ > -T :6082 \ > -f /etc/varnish/default.vcl \ > -p ping_interval=6 -p cli_timeout=10 -p pipe_timeout=600 \ > -p listen_depth=4096 -p thread_pool_min=200 -p > thread_pool_max=500 \ > -p workspace_client=128k -p nuke_limit=1000 \ > -S /etc/varnish/secret \ > -s malloc,6G \ > -s Transient=malloc,3G" > ------- > > Now varnish uses less RAM, and the varnishstats confirm that our limits > should > work: > > ------- > SMA.s0.g_bytes 6.00G -107.61K . 6.00G 6.00G > 6.00G > SMA.s0.g_space 132.70K 107.61K . 136.67K 137.35K > 137.35K > SMA.Transient.g_bytes 1.55G 1022.19 . 1.55G 1.55G > 1.55G > SMA.Transient.g_space 1.45G -1022.19 . 1.45G 1.45G > 1.45G > ------- > > However, varnish, in total, uses up to 12GB RAM instead of only 6GB > (cache) + > 3GB (transient). I tried to find a value in the varnishstat output which > might > indicate how this additional RAM is used, but didn't find anything useful > yet. > > So two questions: > > 1) What might cause varnish to consume considerably more RAM than (cache + > transient > storage)? > > 2) What objects exactly are stored in the transient storage? The > documentation > mentions "shortlived" objects [1] (the "shortlived" parameter is 200 in > our > varnish, which seems to be the varnish debian package default, since we > didn't set this explicitly), but I am not sure if that is limited to > cacheable objects or or not. Also I don't know how to determine which > requests lead to excessive usage of the transient storage. > > Thanks in advance for any further input. > > Cheers, > Marco > > [1] > https://varnish-cache.org/docs/trunk/users-guide/storage-backends.html#transient-storage > > -- > Mit freundlichen Gr??en > Marco Dickert > > Administration und Technik > evolver services GmbH > > Fon +49 / (0)3 71 / 4 00 03 78 24 > Fax +49 / (0)3 71 / 4 00 03 79 > > E-Mail marco.dickert at evolver.de > Web https://www.evolver.de > > Sitz der Gesellschaft: Chemnitz > Handelsregister: Amtsgericht Chemnitz, HRB 22649 > Gesch?ftsf?hrer: Torsten Gramann und Mathias M?ckel > _______________________________________________ > varnish-misc mailing list > varnish-misc at varnish-cache.org > https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc > -------------- next part -------------- An HTML attachment was scrubbed... URL: