Dramatic memory use increase on varnish upgrade
sweeks at imvu.com
Tue Jan 5 06:44:29 CET 2016
We change very few settings in sysctl, and they're all the same
changes from default on both hosts, but here's what everything's
currently set to. Maybe some defaults have changed. I read through
the diff, but nothing jumped out at me as plausibly relevant:
sysctl on a 3.2 kernel host with constant memory use: http://sprunge.us/ThQV
sysctl on a 3.19 kernel host with excessively-growing memory use:
Diff of the two reports: http://sprunge.us/BfMY
All of our deployments so far are on Linux, and nobody here has any
experience with FreeBSD, so adding a new platform to support would be
a pretty big investment, but it might be worth investigating if it
would help get a kernel developer's attention.
On Mon, Jan 4, 2016 at 7:49 PM, Krishna Kumar (Engineering)
<krishna.ku at flipkart.com> wrote:
> How about compare the sysctl settings from the two systems?
> On Tue, Jan 5, 2016 at 8:14 AM, Mark Felder <feld at feld.me> wrote:
>> On Mon, Jan 4, 2016, at 20:24, Mark Felder wrote:
>> > On Mon, Jan 4, 2016, at 19:16, Stephen Weeks wrote:
>> > > We have several varnishd instances that have been running fine for
>> > > years at constant memory use. We've recently decided to upgrade our
>> > > linux distribution. After some troubleshooting, we were unable to
>> > > find a configuration of Varnish on the new distro version that did not
>> > > have dramatically higher memory usage.
>> > I suspect Linux kernel's memory management changes are your problem, not
>> > Varnish.
>> > What is the version of the kernel on the instances that work well for
>> > you?
>> I don't have any expertise on kernels that new. My Linux knowledge ends
>> around 2.6.32 :-)
>> I'd probably look at Linux commits in git under mm/ which is where all
>> the memory management code is at and read through until I saw something
>> significant enough like some new knobs or a major overhaul that might be
>> worth trying to back out and test with a custom kernel.
>> If you have the time and the hardware I'd also try installing FreeBSD
>> and testing your varnish config just to see another data point on memory
>> usage. If it's considerably better, see if you can get a Linux kernel
>> developer's attention and point out that 3.2 and a different OS both
>> fare much, much better for your workload. They might have enough
>> expertise to track down a regression.
>> Mark Felder
>> feld at feld.me
>> varnish-misc mailing list
>> varnish-misc at varnish-cache.org
More information about the varnish-misc