Dramatic memory use increase on varnish upgrade

[Stephen Weeks]

We change very few settings in sysctl, and they're all the same
changes from default on both hosts, but here's what everything's
currently set to.  Maybe some defaults have changed.  I read through
the diff, but nothing jumped out at me as plausibly relevant:

sysctl on a 3.2 kernel host with constant memory use: http://sprunge.us/ThQV
sysctl on a 3.19 kernel host with excessively-growing memory use:
http://sprunge.us/EIcB
Diff of the two reports: http://sprunge.us/BfMY

All of our deployments so far are on Linux, and nobody here has any
experience with FreeBSD, so adding a new platform to support would be
a pretty big investment, but it might be worth investigating if it
would help get a kernel developer's attention.

On Mon, Jan 4, 2016 at 7:49 PM, Krishna Kumar (Engineering)
<krishna.ku at flipkart.com> wrote:
> How about compare the sysctl settings from the two systems?
>
> On Tue, Jan 5, 2016 at 8:14 AM, Mark Felder <feld at feld.me> wrote:
>>
>>
>>
>> On Mon, Jan 4, 2016, at 20:24, Mark Felder wrote:
>> >
>> >
>> > On Mon, Jan 4, 2016, at 19:16, Stephen Weeks wrote:
>> > > We have several varnishd instances that have been running fine for
>> > > years at constant memory use.  We've recently decided to upgrade our
>> > > linux distribution.  After some troubleshooting, we were unable to
>> > > find a configuration of Varnish on the new distro version that did not
>> > > have dramatically higher memory usage.
>> >
>> > I suspect Linux kernel's memory management changes are your problem, not
>> > Varnish.
>> >
>> > What is the version of the kernel on the instances that work well for
>> > you?
>> >
>> >
>>
>> I don't have any expertise on kernels that new. My Linux knowledge ends
>> around 2.6.32 :-)
>>
>> I'd probably look at Linux commits in git under mm/ which is where all
>> the memory management code is at and read through until I saw something
>> significant enough like some new knobs or a major overhaul that might be
>> worth trying to back out and test with a custom kernel.
>>
>> If you have the time and the hardware I'd also try installing FreeBSD
>> and testing your varnish config just to see another data point on memory
>> usage. If it's considerably better, see if you can get a Linux kernel
>> developer's attention and point out that 3.2 and a different OS both
>> fare much, much better for your workload. They might have enough
>> expertise to track down a regression.
>>
>> --
>>   Mark Felder
>>   feld at feld.me
>>
>> _______________________________________________
>> varnish-misc mailing list
>> varnish-misc at varnish-cache.org
>> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
>
>