<div dir="ltr">This is a known and unfortunate issue with the latest versions of jemalloc and certain allocation patterns. You need to downgrade to jemalloc 3.6.<div><br></div><div><a href="https://github.com/varnishcache/varnish-cache/issues/3511#issuecomment-771592238">https://github.com/varnishcache/varnish-cache/issues/3511#issuecomment-771592238</a></div><div><br clear="all"><div><div dir="ltr" class="gmail_signature" data-smartmail="gmail_signature"><div dir="ltr"><div><div dir="ltr">--<br>Reza Naghibi</div><div dir="ltr">VP of Technology<br>Varnish Software</div></div></div></div></div><br></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Wed, May 5, 2021 at 5:35 AM Marco Dickert - evolver group <<a href="mailto:marco.dickert@evolver.de">marco.dickert@evolver.de</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">Hi Geoff,<br>
<br>
thanks for your answer!<br>
<br>
On 2021-04-29 18:00:39, Geoff Simmons wrote:<br>
> You can also turn off THP system-wide (for all processes):<br>
> $ echo never > /sys/kernel/mm/transparent_hugepage/enabled<br>
> That might be your only option if the jemalloc version is too old.<br>
<br>
We diabled huge pages in the kernel, but this didn't solve the problem, the RAM<br>
consumption was unaffected.<br>
<br>
However, we found that the "transient storage" of varnish may be part of our<br>
problem. At least, we could mitigate this behaviour by limiting the transient<br>
storage in the start parameters (last option):<br>
<br>
-------<br>
DAEMON_OPTS="-a :6081 \<br>
-T :6082 \<br>
-f /etc/varnish/default.vcl \<br>
-p ping_interval=6 -p cli_timeout=10 -p pipe_timeout=600 \<br>
-p listen_depth=4096 -p thread_pool_min=200 -p thread_pool_max=500 \<br>
-p workspace_client=128k -p nuke_limit=1000 \<br>
-S /etc/varnish/secret \<br>
-s malloc,6G \<br>
-s Transient=malloc,3G"<br>
-------<br>
<br>
Now varnish uses less RAM, and the varnishstats confirm that our limits should<br>
work:<br>
<br>
-------<br>
SMA.s0.g_bytes 6.00G -107.61K . 6.00G 6.00G 6.00G<br>
SMA.s0.g_space 132.70K 107.61K . 136.67K 137.35K 137.35K<br>
SMA.Transient.g_bytes 1.55G 1022.19 . 1.55G 1.55G 1.55G<br>
SMA.Transient.g_space 1.45G -1022.19 . 1.45G 1.45G 1.45G<br>
-------<br>
<br>
However, varnish, in total, uses up to 12GB RAM instead of only 6GB (cache) +<br>
3GB (transient). I tried to find a value in the varnishstat output which might<br>
indicate how this additional RAM is used, but didn't find anything useful yet.<br>
<br>
So two questions:<br>
<br>
1) What might cause varnish to consume considerably more RAM than (cache + transient<br>
storage)? <br>
<br>
2) What objects exactly are stored in the transient storage? The documentation<br>
mentions "shortlived" objects [1] (the "shortlived" parameter is 200 in our<br>
varnish, which seems to be the varnish debian package default, since we<br>
didn't set this explicitly), but I am not sure if that is limited to<br>
cacheable objects or or not. Also I don't know how to determine which<br>
requests lead to excessive usage of the transient storage.<br>
<br>
Thanks in advance for any further input.<br>
<br>
Cheers,<br>
Marco<br>
<br>
[1] <a href="https://varnish-cache.org/docs/trunk/users-guide/storage-backends.html#transient-storage" rel="noreferrer" target="_blank">https://varnish-cache.org/docs/trunk/users-guide/storage-backends.html#transient-storage</a><br>
<br>
-- <br>
Mit freundlichen Grüßen<br>
Marco Dickert<br>
<br>
Administration und Technik<br>
evolver services GmbH<br>
<br>
Fon +49 / (0)3 71 / 4 00 03 78 24<br>
Fax +49 / (0)3 71 / 4 00 03 79<br>
<br>
E-Mail <a href="mailto:marco.dickert@evolver.de" target="_blank">marco.dickert@evolver.de</a><br>
Web <a href="https://www.evolver.de" rel="noreferrer" target="_blank">https://www.evolver.de</a><br>
<br>
Sitz der Gesellschaft: Chemnitz<br>
Handelsregister: Amtsgericht Chemnitz, HRB 22649<br>
Geschäftsführer: Torsten Gramann und Mathias Möckel<br>
_______________________________________________<br>
varnish-misc mailing list<br>
<a href="mailto:varnish-misc@varnish-cache.org" target="_blank">varnish-misc@varnish-cache.org</a><br>
<a href="https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc" rel="noreferrer" target="_blank">https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc</a><br>
</blockquote></div>