Transparent hugepages on RHEL8

[Geoff Simmons]

On 5/24/20 01:29, info+varnish at shee.org wrote:
> This notes 
> 
> https://varnish-cache.org/docs/trunk/installation/platformnotes.html
> 
> has a comment about "Transparent hugepages".
> 
> Does this still apply to EL8?

That's a good heads-up that those docs need to be updated -- they refer
to RHEL6 and Linux kernel 3.2. If I'm not mistaken, enabling THP by
default was fairly new at the time, but it's still the default and
that's old news now, as your settings confirmed (just checked that it's
also the default on my Debian stretch laptop).

The issue is not really the distro or kernel version, but the use of the
THP feature, and it's still a problem, probably always will be. AFAICT
THP does nothing good for Varnish. It's harmless if you're lucky, but it
can be very disruptive.

I haven't tried it with RHEL8. The doc says that it "is known to cause
sporadic crashes of Varnish", but while I haven't seen crashes, on RHEL7
I've seen that the memory usage of the cache process bloats up
enormously, orders of magnitude larger than the actual size of the cache
and anything else in Varnish that occupies memory. After disabling THP
for Varnish (as detailed below), I saw memory usage become much smaller,
more like what you'd expect from the cache size and other overhead.

There's an explanation for why THP causes that, but suffice it to say
that THP creates trouble for a variety of apps that manage a lot of
memory. MongoDB, Oracle, redis and many other projects advise you to
turn it off. THP is inevitably a problem for the jemalloc memory
allocator, which is invariably used with Varnish.

You can turn off THP system-wide with:

$ echo never > /sys/kernel/mm/transparent_hugepage/enabled

Or, I believe this may work in /etc/grub.conf:

transparent_hugepage=never

Since you're on RHEL8, you also have the option of disabling THP for
jemalloc when used by Varnish; so you don't have to turn it off for
everything, if you prefer to leave on the default for other processes on
your system. The option is: thp:never

One way to do that is to start Varnish with that setting in MALLOC_CONF
in its environment:

$ MALLOC_CONF=thp:never /usr/sbin/varnishd -a :80 ...

Or you could set thp:never in /etc/malloc.conf, in which case the
setting holds for any app that uses jemalloc. The jemalloc man page has
all the details.

This is possible in RHEL8 because el8 supports versions of jemalloc that
have the option. Earlier versions of jemalloc didn't have it, in
particular 3.0.6, on which the world was stuck for a very long time, and
was the latest available in el7.

For readers who are using el7/RHEL7 -- I patched up an RPM that installs
more recent jemalloc on el7. Git repo here:

https://code.uplex.de/uplex-varnish/libjemalloc2-el7-rpm
(Still no README there, my bad.)

There's an el7 package repo with the RPM at https://pkg.uplex.de/:

$ yum-config-manager --add-repo
https://pkg.uplex.de/rpm/7/uplex-varnish/x86_64/

Newer versions of jemalloc have a different SO name, libjemalloc.so.2,
whereas most software built for el7 that links to jemalloc expects
libjemalloc.so.1. That includes the el7 Varnish RPMs from packagecloud.
So if you have that in your Varnish binary (if ldd points to
libjemalloc.so.1), and you want to use the el7 RPM for newer jemalloc,
you can do this:

$ patchelf --replace-needed libjemalloc.so.1 libjemalloc.so.2
/path/to/varnishd

Then check ldd, you should see it now pointing to libjemalloc.so.2.

I have that working in production, which made the thp:never setting
possible, and that got rid of the memory bloat.

Sorry for the long-winded response, I had a big fight with this problem
a few months ago, and am still a little miffed at transparent hugepages.


HTH,
Geoff


PS while I'm on the subject, shout-out to Ingvar Hagelund, who does the
epel/fedora packaging for Varnish, jemalloc, and a variety of other
things. He did the hard work packaging jemalloc, I just changed a
version number.

-- 
** * * UPLEX - Nils Goroll Systemoptimierung

Scheffelstraße 32
22301 Hamburg

Tel +49 40 2880 5731
Mob +49 176 636 90917
Fax +49 40 42949753

http://uplex.de

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 833 bytes
Desc: OpenPGP digital signature
URL: <https://www.varnish-cache.org/lists/pipermail/varnish-misc/attachments/20200524/c45b5bee/attachment.bin>