Dramatic memory use increase on varnish upgrade

[Stephen Weeks]

We have several varnishd instances that have been running fine for
years at constant memory use.  We've recently decided to upgrade our
linux distribution.  After some troubleshooting, we were unable to
find a configuration of Varnish on the new distro version that did not
have dramatically higher memory usage.  After that, we tried upgrading
to a modern, supported version of Varnish, but were also unable to
find a configuration that doesn't use dramatically more memory.  We
have tried the following configurations:

* Varnish 3.0.3, Ubuntu 10.04, Linux 3.2, 24G malloc storage, glibc
malloc, constant memory use at 29G rss
* Varnish 3.0.5, Ubuntu 14.04, Linux 3.19, 24G malloc storage,
jemalloc, memory usage grows until it passes 64G and starts swapping
* Varnish 4.1.0, Ubuntu 14.04, Linux 3.19, 24G malloc storage,
jemalloc, memory usage grows until it passes 64G and starts swapping
* Varnish 4.1.0, Ubuntu 14.04, Linux 3.19, 24G malloc storage, glibc
malloc, memory usage grows faster than with jemalloc until it passes
64G and starts swapping

I've reviewed varnishstat output, but I've been unable to account for
the increased memory usage.  I asked the IRC channel for support a few
weeks ago, and nobody else there was able to account for this in
varnishstat, and the only idea that we were able to come up with was
to try glibc malloc with Varnish 4.1, which as you can see above
turned out to be slightly worse.  I'm really out of ideas here; any
help in trying to find a configuration that has constrained memory
usage on a more-modern platform would be highly appreciated.

Details on the functional Varnish 3 instances with a steady 29G memory usage:

Command line: /usr/sbin/varnishd -P /var/run/varnishd.pid -a :3000 -T
localhost:6082 -f /etc/varnish/default.vcl -S /etc/varnish/secret -p
sess_timeout 60 -p thread_pool_min 500 -p thread_pool_max 2000 -p
nuke_limit 1000 -s malloc,24G
varnishstat -1: http://sprunge.us/iKUC
vcl config: http://sprunge.us/LOUE
libraries linked: http://sprunge.us/FOSL

Details on the Varnish 3 instances with much higher memory usage:

Command line: /usr/sbin/varnishd -P /var/run/varnishd.pid -a :3000 -T
localhost:6082 -f /etc/varnish/default.vcl -S /etc/varnish/secret -p
sess_timeout=60 -p thread_pool_min=500 -p thread_pool_max=2000 -p
nuke_limit=1000 -s malloc,24G
varnishstat -1: http://sprunge.us/KGIa
vcl config: Same config as above, http://sprunge.us/LOUE

Details on the varnish 4.1 instances with much higher memory usage:

Command line: /usr/sbin/varnishd -P /run/varnishd.pid -a :3000 -T
localhost:6082 -f /etc/varnish/default.vcl -S /etc/varnish/secret -p
timeout_idle=60 -p thread_pool_min=500 -p thread_pool_max=2000 -p
nuke_limit=1000 -s malloc,24G
varnishstat -1: http://sprunge.us/gCOG
vcl config: http://sprunge.us/LQRM
libraries linked: http://sprunge.us/WCbj

Graph of memory usage for two varnish instances.  AF002245 is Varnish
3.0.3, AF002256 is Varnish 4.1.0: http://i.imgur.com/Fxsms48.png

The closest I've been able to find on Google is people talking about
accidentally using too much Transient space, but in varnishstat I see
<800k used in transient:

SMA.Transient.g_bytes                   766728          .   Bytes outstanding

They both have comparable numbers of objects cached:

n_object                419597          .   N struct object
MAIN.n_object                420371          .   object structs made

I'm pretty sure I've got to have something bad in my config, because
I'm seeing the same behaviour on both 3.0.5 and 4.1.0, and a memory
leak this bad for so long seems very unlikely.  I've considered trying
to narrow this down by porting forward exactly the same 3.0.3 version
to a slightly more modern platform, or trying to backport 4.1.0 back
to the ancient Ubuntu 10.04, but I'm not really sure what I'd do
differently if either of those worked, as I would rather not stay on
such an old platform if I can avoid it, in either case.

Are there any other metrics I can gather here?  Can anyone give me any
recommendations on how I can configure Varnish to constrain its memory
usage?  Any ideas about what I'm missing?