Transparent hugepages on RHEL8
info+varnish at shee.org
info+varnish at shee.org
Mon May 25 09:26:40 UTC 2020
Am Sonntag, den 24.05.2020, 10:09 +0200 schrieb Geoff Simmons:
> On 5/24/20 01:29, info+varnish at shee.org wrote:
> > This notes
> > https://varnish-cache.org/docs/trunk/installation/platformnotes.html
> > has a comment about "Transparent hugepages".
> > Does this still apply to EL8?
> That's a good heads-up that those docs need to be updated -- they
> to RHEL6 and Linux kernel 3.2. If I'm not mistaken, enabling THP by
> default was fairly new at the time, but it's still the default and
> that's old news now, as your settings confirmed (just checked that
> also the default on my Debian stretch laptop).
> The issue is not really the distro or kernel version, but the use of
> THP feature, and it's still a problem, probably always will be.
> THP does nothing good for Varnish. It's harmless if you're lucky, but
> can be very disruptive.
> I haven't tried it with RHEL8. The doc says that it "is known to
> sporadic crashes of Varnish", but while I haven't seen crashes, on
> I've seen that the memory usage of the cache process bloats up
> enormously, orders of magnitude larger than the actual size of the
> and anything else in Varnish that occupies memory. After disabling
> for Varnish (as detailed below), I saw memory usage become much
> more like what you'd expect from the cache size and other overhead.
> There's an explanation for why THP causes that, but suffice it to say
> that THP creates trouble for a variety of apps that manage a lot of
> memory. MongoDB, Oracle, redis and many other projects advise you to
> turn it off. THP is inevitably a problem for the jemalloc memory
> allocator, which is invariably used with Varnish.
> You can turn off THP system-wide with:
> $ echo never > /sys/kernel/mm/transparent_hugepage/enabled
> Or, I believe this may work in /etc/grub.conf:
> Since you're on RHEL8, you also have the option of disabling THP for
> jemalloc when used by Varnish; so you don't have to turn it off for
> everything, if you prefer to leave on the default for other processes
> your system. The option is: thp:never
> One way to do that is to start Varnish with that setting in
> in its environment:
> $ MALLOC_CONF=thp:never /usr/sbin/varnishd -a :80 ...
> Or you could set thp:never in /etc/malloc.conf, in which case the
> setting holds for any app that uses jemalloc. The jemalloc man page
> all the details.
> This is possible in RHEL8 because el8 supports versions of jemalloc
> have the option. Earlier versions of jemalloc didn't have it, in
> particular 3.0.6, on which the world was stuck for a very long time,
> was the latest available in el7.
> For readers who are using el7/RHEL7 -- I patched up an RPM that
> more recent jemalloc on el7. Git repo here:
> (Still no README there, my bad.)
> There's an el7 package repo with the RPM at https://pkg.uplex.de/:
> $ yum-config-manager --add-repo
> Newer versions of jemalloc have a different SO name,
> whereas most software built for el7 that links to jemalloc expects
> libjemalloc.so.1. That includes the el7 Varnish RPMs from
> So if you have that in your Varnish binary (if ldd points to
> libjemalloc.so.1), and you want to use the el7 RPM for newer
> you can do this:
> $ patchelf --replace-needed libjemalloc.so.1 libjemalloc.so.2
> Then check ldd, you should see it now pointing to libjemalloc.so.2.
> I have that working in production, which made the thp:never setting
> possible, and that got rid of the memory bloat.
> Sorry for the long-winded response, I had a big fight with this
> a few months ago, and am still a little miffed at transparent
Danke Geoff! for your explanatory answer. It gives some insights
that motivates to go further then just disabling a system option.
I will take a deeper dive into the current context (EL8) and check
what is still relevant and what not and then come back ...
More information about the varnish-misc