Dramatic memory use increase on varnish upgrade

[Mark Felder]

On Mon, Jan 4, 2016, at 20:24, Mark Felder wrote:
> 
> 
> On Mon, Jan 4, 2016, at 19:16, Stephen Weeks wrote:
> > We have several varnishd instances that have been running fine for
> > years at constant memory use.  We've recently decided to upgrade our
> > linux distribution.  After some troubleshooting, we were unable to
> > find a configuration of Varnish on the new distro version that did not
> > have dramatically higher memory usage.  
> 
> I suspect Linux kernel's memory management changes are your problem, not
> Varnish.
> 
> What is the version of the kernel on the instances that work well for
> you?
> 
> 

I don't have any expertise on kernels that new. My Linux knowledge ends
around 2.6.32 :-)

I'd probably look at Linux commits in git under mm/ which is where all
the memory management code is at and read through until I saw something
significant enough like some new knobs or a major overhaul that might be
worth trying to back out and test with a custom kernel.

If you have the time and the hardware I'd also try installing FreeBSD
and testing your varnish config just to see another data point on memory
usage. If it's considerably better, see if you can get a Linux kernel
developer's attention and point out that 3.2 and a different OS both
fare much, much better for your workload. They might have enough
expertise to track down a regression.

-- 
  Mark Felder
  feld at feld.me