Transparent hugepages on RHEL8

[info+varnish at shee.org]

Am Sonntag, den 24.05.2020, 10:09 +0200 schrieb Geoff Simmons:
> On 5/24/20 01:29, info+varnish at shee.org wrote:
> > This notes 
> > 
> > https://varnish-cache.org/docs/trunk/installation/platformnotes.html
> > 
> > has a comment about "Transparent hugepages".
> > 
> > Does this still apply to EL8?
> 
> That's a good heads-up that those docs need to be updated -- they
> refer
> to RHEL6 and Linux kernel 3.2. If I'm not mistaken, enabling THP by
> default was fairly new at the time, but it's still the default and
> that's old news now, as your settings confirmed (just checked that
> it's
> also the default on my Debian stretch laptop).
> 
> The issue is not really the distro or kernel version, but the use of
> the
> THP feature, and it's still a problem, probably always will be.
> AFAICT
> THP does nothing good for Varnish. It's harmless if you're lucky, but
> it
> can be very disruptive.
> 
> I haven't tried it with RHEL8. The doc says that it "is known to
> cause
> sporadic crashes of Varnish", but while I haven't seen crashes, on
> RHEL7
> I've seen that the memory usage of the cache process bloats up
> enormously, orders of magnitude larger than the actual size of the
> cache
> and anything else in Varnish that occupies memory. After disabling
> THP
> for Varnish (as detailed below), I saw memory usage become much
> smaller,
> more like what you'd expect from the cache size and other overhead.
> 
> There's an explanation for why THP causes that, but suffice it to say
> that THP creates trouble for a variety of apps that manage a lot of
> memory. MongoDB, Oracle, redis and many other projects advise you to
> turn it off. THP is inevitably a problem for the jemalloc memory
> allocator, which is invariably used with Varnish.
> 
> You can turn off THP system-wide with:
> 
> $ echo never > /sys/kernel/mm/transparent_hugepage/enabled
> 
> Or, I believe this may work in /etc/grub.conf:
> 
> transparent_hugepage=never
> 
> Since you're on RHEL8, you also have the option of disabling THP for
> jemalloc when used by Varnish; so you don't have to turn it off for
> everything, if you prefer to leave on the default for other processes
> on
> your system. The option is: thp:never
> 
> One way to do that is to start Varnish with that setting in
> MALLOC_CONF
> in its environment:
> 
> $ MALLOC_CONF=thp:never /usr/sbin/varnishd -a :80 ...
> 
> Or you could set thp:never in /etc/malloc.conf, in which case the
> setting holds for any app that uses jemalloc. The jemalloc man page
> has
> all the details.
> 
> This is possible in RHEL8 because el8 supports versions of jemalloc
> that
> have the option. Earlier versions of jemalloc didn't have it, in
> particular 3.0.6, on which the world was stuck for a very long time,
> and
> was the latest available in el7.
> 
> For readers who are using el7/RHEL7 -- I patched up an RPM that
> installs
> more recent jemalloc on el7. Git repo here:
> 
> https://code.uplex.de/uplex-varnish/libjemalloc2-el7-rpm
> (Still no README there, my bad.)
> 
> There's an el7 package repo with the RPM at https://pkg.uplex.de/:
> 
> $ yum-config-manager --add-repo
> https://pkg.uplex.de/rpm/7/uplex-varnish/x86_64/
> 
> Newer versions of jemalloc have a different SO name,
> libjemalloc.so.2,
> whereas most software built for el7 that links to jemalloc expects
> libjemalloc.so.1. That includes the el7 Varnish RPMs from
> packagecloud.
> So if you have that in your Varnish binary (if ldd points to
> libjemalloc.so.1), and you want to use the el7 RPM for newer
> jemalloc,
> you can do this:
> 
> $ patchelf --replace-needed libjemalloc.so.1 libjemalloc.so.2
> /path/to/varnishd
> 
> Then check ldd, you should see it now pointing to libjemalloc.so.2.
> 
> I have that working in production, which made the thp:never setting
> possible, and that got rid of the memory bloat.
> 
> Sorry for the long-winded response, I had a big fight with this
> problem
> a few months ago, and am still a little miffed at transparent
> hugepages.
> 


Danke Geoff! for your explanatory answer. It gives some insights
that motivates to go further then just disabling a system option. 

I will take a deeper dive into the current context (EL8) and check
what is still relevant and what not and then come back ...


--
Leon