Varnish suddenly started using much more memory
Guillaume Quintard
guillaume.quintard at gmail.com
Sat May 18 18:01:49 UTC 2024
Sorry, I should have been clearer, I meant: where are the varnish packages
coming from? Are they from the official repositories, from
https://packagecloud.io/varnishcache/ or built from source maybe?
If you don't have old metrics (you should really invest some time in
something like prometheus, it would probably have made the issue obvious),
then we can't really compare anything. Is there any chance you can run the
old version on the server to explore the differences?
Two extra questions:
- what's the output of: varnishstat -1 -f '*g_bytes'
- have you tweaked any workspaces/thread parameters?
Cheers,
On Fri, May 17, 2024, 06:17 Batanun B <batanun at hotmail.com> wrote:
> Hi,
>
> Naturally, I can't be certain that the "in my mind" trivial VCL changes
> can't be the culprit. But I just can't see the logic in those changes
> causing this massive change in memory usage. But I'll summarize the changes
> here, and maybe you can identify a suspect:
>
> * Modified the xkey header used by the xkey vmod, adding the id of the
> current website
> * Modified the TTL, from 1w to 1h, for a specific type of resource
> existing in maybe 20 versions (ie different urls), each being about 5 kB in
> size
> * Modified the backend probe url, from the startpage (ie full html) to a
> dedicated healthcheck endpoint (much smaller footprint, and much quicker)
>
> That's it. That's all the VCL changes we made in that deployment. And,
> like I said, we did no changes in the OS or Varnish config.
>
>
> > check the difference in passes, if they are about the same, look for
> hit-for-misses,
>
> We don't have those statistics from the old server, so I can't do a
> comparison. But here are the current statistics:
>
> MAIN.s_pass 180721 0.14 Total pass-ed
> requests seen
> MAIN.cache_hitpass 0 0.00 Cache hits for pass.
> MAIN.cache_hit 3718468 2.86 Cache hits
> MAIN.cache_hit_grace 53903 0.04 Cache grace hits
> MAIN.cache_hitpass 0 0.00 Cache hits for pass.
> MAIN.cache_hitmiss 1129 0.00 Cache hits for miss.
>
>
> > and lastly, look at how long Varnish is trying to cache the average
> object.
>
> I'm not sure how I do that. Is there a varnishstat counter I can look at?
>
> > which packages are you using?
>
> Instead of giving you the full list, I guess it makes more sense to just
> list the one that differ. Below is a diff output of "apt list --installed"
> of before and after the deploy.
>
> Regards
>
>
> 2c2
> < accountsservice/now 0.6.55-0ubuntu12~20.04.5 amd64 [installed,upgradable
> to: 0.6.55-0ubuntu12~20.04.7]
> ---
> > accountsservice/focal-updates,focal-security,now
> 0.6.55-0ubuntu12~20.04.7 amd64 [installed,automatic]
> 23,25c23,25
> < bind9-dnsutils/now 1:9.16.1-0ubuntu2.12 amd64 [installed,upgradable to:
> 1:9.16.48-0ubuntu0.20.04.1]
> < bind9-host/now 1:9.16.1-0ubuntu2.12 amd64 [installed,upgradable to:
> 1:9.16.48-0ubuntu0.20.04.1]
> < bind9-libs/now 1:9.16.1-0ubuntu2.12 amd64 [installed,upgradable to:
> 1:9.16.48-0ubuntu0.20.04.1]
> ---
> > bind9-dnsutils/focal-updates,focal-security,now
> 1:9.16.48-0ubuntu0.20.04.1 amd64 [installed,automatic]
> > bind9-host/focal-updates,focal-security,now 1:9.16.48-0ubuntu0.20.04.1
> amd64 [installed,automatic]
> > bind9-libs/focal-updates,focal-security,now 1:9.16.48-0ubuntu0.20.04.1
> amd64 [installed,automatic]
> 31c31
> < bsdutils/focal-security,now 1:2.34-0.1ubuntu9.3 amd64
> [installed,upgradable to: 1:2.34-0.1ubuntu9.4]
> ---
> > bsdutils/now 1:2.34-0.1ubuntu9.3 amd64 [installed,upgradable to:
> 1:2.34-0.1ubuntu9.6]
> 41c41
> < cloud-init/now 22.4.2-0ubuntu0~20.04.2 all [installed,upgradable to:
> 23.4.4-0ubuntu0~20.04.1]
> ---
> > cloud-init/now 22.4.2-0ubuntu0~20.04.2 all [installed,upgradable to:
> 24.1.3-0ubuntu1~20.04.1]
> 48c48
> < cpio/focal-updates,focal-security,now 2.13+dfsg-2ubuntu0.3 amd64
> [installed,automatic]
> ---
> > cpio/now 2.13+dfsg-2ubuntu0.3 amd64 [installed,upgradable to:
> 2.13+dfsg-2ubuntu0.4]
> 56c56
> < curl/focal-updates,focal-security,now 7.68.0-1ubuntu2.21 amd64
> [installed]
> ---
> > curl/focal-updates,focal-security,now 7.68.0-1ubuntu2.22 amd64
> [installed]
> 68c68
> < distro-info-data/now 0.43ubuntu1.11 all [installed,upgradable to:
> 0.43ubuntu1.15]
> ---
> > distro-info-data/now 0.43ubuntu1.11 all [installed,upgradable to:
> 0.43ubuntu1.16]
> 82c82
> < fdisk/focal-security,now 2.34-0.1ubuntu9.3 amd64 [installed,upgradable
> to: 2.34-0.1ubuntu9.4]
> ---
> > fdisk/now 2.34-0.1ubuntu9.3 amd64 [installed,upgradable to:
> 2.34-0.1ubuntu9.6]
> 119,120c119,120
> < grub-efi-amd64-bin/now 2.06-2ubuntu14.1 amd64 [installed,upgradable to:
> 2.06-2ubuntu14.4]
> < grub-efi-amd64-signed/now 1.187.3~20.04.1+2.06-2ubuntu14.1 amd64
> [installed,upgradable to: 1.187.6~20.04.1+2.06-2ubuntu14.4]
> ---
> > grub-efi-amd64-bin/focal-updates,focal-security,now 2.06-2ubuntu14.4
> amd64 [installed]
> > grub-efi-amd64-signed/focal-updates,focal-security,now
> 1.187.6~20.04.1+2.06-2ubuntu14.4 amd64 [installed]
> 148c148
> < klibc-utils/focal-updates,focal-security,now 2.0.7-1ubuntu5.1 amd64
> [installed,automatic]
> ---
> > klibc-utils/focal-updates,focal-security,now 2.0.7-1ubuntu5.2 amd64
> [installed,automatic]
> 152c152
> < landscape-common/focal-updates,now 19.12-0ubuntu4.3 amd64
> [installed,automatic]
> ---
> > landscape-common/now 19.12-0ubuntu4.3 amd64 [installed,upgradable to:
> 23.02-0ubuntu1~20.04.2]
> 154,155c154,155
> < less/now 551-1ubuntu0.1 amd64 [installed,upgradable to: 551-1ubuntu0.2]
> < libaccountsservice0/now 0.6.55-0ubuntu12~20.04.5 amd64
> [installed,upgradable to: 0.6.55-0ubuntu12~20.04.7]
> ---
> > less/now 551-1ubuntu0.1 amd64 [installed,upgradable to: 551-1ubuntu0.3]
> > libaccountsservice0/focal-updates,focal-security,now
> 0.6.55-0ubuntu12~20.04.7 amd64 [installed,automatic]
> 175c175
> < libblkid1/focal-security,now 2.34-0.1ubuntu9.3 amd64
> [installed,upgradable to: 2.34-0.1ubuntu9.4]
> ---
> > libblkid1/now 2.34-0.1ubuntu9.3 amd64 [installed,upgradable to:
> 2.34-0.1ubuntu9.6]
> 187,190c187,190
> < libc-bin/now 2.31-0ubuntu9.9 amd64 [installed,upgradable to:
> 2.31-0ubuntu9.14]
> < libc-dev-bin/focal-updates,focal-security,now 2.31-0ubuntu9.14 amd64
> [installed,automatic]
> < libc6-dev/focal-updates,focal-security,now 2.31-0ubuntu9.14 amd64
> [installed,automatic]
> < libc6/focal-updates,focal-security,now 2.31-0ubuntu9.14 amd64
> [installed,automatic]
> ---
> > libc-bin/now 2.31-0ubuntu9.9 amd64 [installed,upgradable to:
> 2.31-0ubuntu9.15]
> > libc-dev-bin/focal-updates,focal-security,now 2.31-0ubuntu9.15 amd64
> [installed,automatic]
> > libc6-dev/focal-updates,focal-security,now 2.31-0ubuntu9.15 amd64
> [installed,automatic]
> > libc6/focal-updates,focal-security,now 2.31-0ubuntu9.15 amd64
> [installed,automatic]
> 203,204c203,204
> < libcurl3-gnutls/now 7.68.0-1ubuntu2.15 amd64 [installed,upgradable to:
> 7.68.0-1ubuntu2.21]
> < libcurl4/focal-updates,focal-security,now 7.68.0-1ubuntu2.21 amd64
> [installed,automatic]
> ---
> > libcurl3-gnutls/now 7.68.0-1ubuntu2.15 amd64 [installed,upgradable to:
> 7.68.0-1ubuntu2.22]
> > libcurl4/focal-updates,focal-security,now 7.68.0-1ubuntu2.22 amd64
> [installed,automatic]
> 227c227
> < libfdisk1/focal-security,now 2.34-0.1ubuntu9.3 amd64
> [installed,upgradable to: 2.34-0.1ubuntu9.4]
> ---
> > libfdisk1/now 2.34-0.1ubuntu9.3 amd64 [installed,upgradable to:
> 2.34-0.1ubuntu9.6]
> 244,246c244,246
> < libglib2.0-0/now 2.64.6-1~ubuntu20.04.4 amd64 [installed,upgradable to:
> 2.64.6-1~ubuntu20.04.6]
> < libglib2.0-bin/now 2.64.6-1~ubuntu20.04.4 amd64 [installed,upgradable
> to: 2.64.6-1~ubuntu20.04.6]
> < libglib2.0-data/now 2.64.6-1~ubuntu20.04.4 all [installed,upgradable to:
> 2.64.6-1~ubuntu20.04.6]
> ---
> > libglib2.0-0/now 2.64.6-1~ubuntu20.04.4 amd64 [installed,upgradable to:
> 2.64.6-1~ubuntu20.04.7]
> > libglib2.0-bin/now 2.64.6-1~ubuntu20.04.4 amd64 [installed,upgradable
> to: 2.64.6-1~ubuntu20.04.7]
> > libglib2.0-data/now 2.64.6-1~ubuntu20.04.4 all [installed,upgradable to:
> 2.64.6-1~ubuntu20.04.7]
> 248c248
> < libgnutls30/now 3.6.13-2ubuntu1.7 amd64 [installed,upgradable to:
> 3.6.13-2ubuntu1.10]
> ---
> > libgnutls30/now 3.6.13-2ubuntu1.7 amd64 [installed,upgradable to:
> 3.6.13-2ubuntu1.11]
> 278c278
> < libklibc/focal-updates,focal-security,now 2.0.7-1ubuntu5.1 amd64
> [installed,automatic]
> ---
> > libklibc/focal-updates,focal-security,now 2.0.7-1ubuntu5.2 amd64
> [installed,automatic]
> 284c284
> < libldap-2.4-2/now 2.4.49+dfsg-2ubuntu1.9 amd64 [installed,upgradable to:
> 2.4.49+dfsg-2ubuntu1.10]
> ---
> > libldap-2.4-2/focal-updates,focal-security,now 2.4.49+dfsg-2ubuntu1.10
> amd64 [installed,automatic]
> 301c301
> < libmount1/focal-security,now 2.34-0.1ubuntu9.3 amd64
> [installed,upgradable to: 2.34-0.1ubuntu9.4]
> ---
> > libmount1/now 2.34-0.1ubuntu9.3 amd64 [installed,upgradable to:
> 2.34-0.1ubuntu9.6]
> 306,307c306,307
> < libncurses6/focal,now 6.2-0ubuntu2 amd64 [installed,upgradable to:
> 6.2-0ubuntu2.1]
> < libncursesw6/focal,now 6.2-0ubuntu2 amd64 [installed,upgradable to:
> 6.2-0ubuntu2.1]
> ---
> > libncurses6/focal-updates,focal-security,now 6.2-0ubuntu2.1 amd64
> [installed,automatic]
> > libncursesw6/focal-updates,focal-security,now 6.2-0ubuntu2.1 amd64
> [installed,automatic]
> 314c314
> < libnghttp2-14/focal,now 1.40.0-1build1 amd64 [installed,upgradable to:
> 1.40.0-1ubuntu0.2]
> ---
> > libnghttp2-14/focal,now 1.40.0-1build1 amd64 [installed,upgradable to:
> 1.40.0-1ubuntu0.3]
> 316,318c316,318
> < libnspr4/focal,now 2:4.25-1 amd64 [installed,automatic]
> < libnss-systemd/now 245.4-4ubuntu3.19 amd64 [installed,upgradable to:
> 245.4-4ubuntu3.23]
> < libnss3/now 2:3.49.1-1ubuntu1.8 amd64 [installed,upgradable to:
> 2:3.49.1-1ubuntu1.9]
> ---
> > libnspr4/focal,now 2:4.25-1 amd64 [installed,upgradable to:
> 2:4.35-0ubuntu0.20.04.1]
> > libnss-systemd/focal-security,now 245.4-4ubuntu3.20 amd64
> [installed,upgradable to: 245.4-4ubuntu3.23]
> > libnss3/now 2:3.49.1-1ubuntu1.8 amd64 [installed,upgradable to:
> 2:3.98-0ubuntu0.20.04.2]
> 325,328c325,328
> < libpam-modules-bin/now 1.3.1-5ubuntu4.6 amd64 [installed,upgradable to:
> 1.3.1-5ubuntu4.7]
> < libpam-modules/now 1.3.1-5ubuntu4.6 amd64 [installed,upgradable to:
> 1.3.1-5ubuntu4.7]
> < libpam-runtime/now 1.3.1-5ubuntu4.6 all [installed,upgradable to:
> 1.3.1-5ubuntu4.7]
> < libpam-systemd/now 245.4-4ubuntu3.19 amd64 [installed,upgradable to:
> 245.4-4ubuntu3.23]
> ---
> > libpam-modules-bin/focal-updates,focal-security,now 1.3.1-5ubuntu4.7
> amd64 [installed,automatic]
> > libpam-modules/focal-updates,focal-security,now 1.3.1-5ubuntu4.7 amd64
> [installed,automatic]
> > libpam-runtime/focal-updates,focal-security,now 1.3.1-5ubuntu4.7 all
> [installed,automatic]
> > libpam-systemd/focal-security,now 245.4-4ubuntu3.20 amd64
> [installed,upgradable to: 245.4-4ubuntu3.23]
> 347c347
> < libprocps8/now 2:3.3.16-1ubuntu2.3 amd64 [installed,upgradable to:
> 2:3.3.16-1ubuntu2.4]
> ---
> > libprocps8/focal-updates,focal-security,now 2:3.3.16-1ubuntu2.4 amd64
> [installed,automatic]
> 375c375
> < libsmartcols1/focal-security,now 2.34-0.1ubuntu9.3 amd64
> [installed,upgradable to: 2.34-0.1ubuntu9.4]
> ---
> > libsmartcols1/now 2.34-0.1ubuntu9.3 amd64 [installed,upgradable to:
> 2.34-0.1ubuntu9.6]
> 386c386
> < libsystemd0/now 245.4-4ubuntu3.19 amd64 [installed,upgradable to:
> 245.4-4ubuntu3.23]
> ---
> > libsystemd0/focal-security,now 245.4-4ubuntu3.20 amd64
> [installed,upgradable to: 245.4-4ubuntu3.23]
> 394c394
> < libtinfo6/focal,now 6.2-0ubuntu2 amd64 [installed,upgradable to:
> 6.2-0ubuntu2.1]
> ---
> > libtinfo6/focal-updates,focal-security,now 6.2-0ubuntu2.1 amd64
> [installed,automatic]
> 407c407
> < libuuid1/focal-security,now 2.34-0.1ubuntu9.3 amd64
> [installed,upgradable to: 2.34-0.1ubuntu9.4]
> ---
> > libuuid1/now 2.34-0.1ubuntu9.3 amd64 [installed,upgradable to:
> 2.34-0.1ubuntu9.6]
> 434c434
> < linux-azure/now 5.15.0.1033.40~20.04.23 amd64 [installed,upgradable to:
> 5.15.0.1059.67~20.04.49]
> ---
> > linux-azure/now 5.15.0.1033.40~20.04.23 amd64 [installed,upgradable to:
> 5.15.0.1064.73~20.04.1]
> 438,439c438,439
> < linux-cloud-tools-azure/now 5.15.0.1033.40~20.04.23 amd64
> [installed,upgradable to: 5.15.0.1059.67~20.04.49]
> < linux-cloud-tools-common/focal-updates,focal-security,now 5.4.0-139.156
> all [installed,upgradable to: 5.4.0-174.193]
> ---
> > linux-cloud-tools-azure/now 5.15.0.1033.40~20.04.23 amd64
> [installed,upgradable to: 5.15.0.1064.73~20.04.1]
> > linux-cloud-tools-common/focal-updates,focal-security,now 5.4.0-182.202
> all [installed]
> 441c441
> < linux-headers-azure/now 5.15.0.1033.40~20.04.23 amd64
> [installed,upgradable to: 5.15.0.1059.67~20.04.49]
> ---
> > linux-headers-azure/now 5.15.0.1033.40~20.04.23 amd64
> [installed,upgradable to: 5.15.0.1064.73~20.04.1]
> 443,444c443,444
> < linux-image-azure/now 5.15.0.1033.40~20.04.23 amd64
> [installed,upgradable to: 5.15.0.1059.67~20.04.49]
> < linux-libc-dev/focal-updates,focal-security,now 5.4.0-174.193 amd64
> [installed,automatic]
> ---
> > linux-image-azure/now 5.15.0.1033.40~20.04.23 amd64
> [installed,upgradable to: 5.15.0.1064.73~20.04.1]
> > linux-libc-dev/focal-updates,focal-security,now 5.4.0-182.202 amd64
> [installed,automatic]
> 447,449c447,449
> < linux-tools-azure/now 5.15.0.1033.40~20.04.23 amd64
> [installed,upgradable to: 5.15.0.1059.67~20.04.49]
> < linux-tools-common/focal-updates,focal-security,now 5.4.0-139.156 all
> [installed,upgradable to: 5.4.0-174.193]
> < locales/now 2.31-0ubuntu9.9 all [installed,upgradable to:
> 2.31-0ubuntu9.14]
> ---
> > linux-tools-azure/now 5.15.0.1033.40~20.04.23 amd64
> [installed,upgradable to: 5.15.0.1064.73~20.04.1]
> > linux-tools-common/focal-updates,focal-security,now 5.4.0-139.156 all
> [installed,upgradable to: 5.4.0-182.202]
> > locales/now 2.31-0ubuntu9.9 all [installed,upgradable to:
> 2.31-0ubuntu9.15]
> 473c473
> < mount/focal-security,now 2.34-0.1ubuntu9.3 amd64 [installed,upgradable
> to: 2.34-0.1ubuntu9.4]
> ---
> > mount/focal-updates,focal-security,now 2.34-0.1ubuntu9.6 amd64
> [installed,automatic]
> 499c499
> < passwd/now 1:4.8.1-1ubuntu5.20.04.4 amd64 [installed,upgradable to:
> 1:4.8.1-1ubuntu5.20.04.5]
> ---
> > passwd/focal-updates,focal-security,now 1:4.8.1-1ubuntu5.20.04.5 amd64
> [installed,automatic]
> 555c555
> < python3-jinja2/focal,now 2.10.1-2 all [installed,upgradable to:
> 2.10.1-2ubuntu0.2]
> ---
> > python3-jinja2/focal-updates,focal-security,now 2.10.1-2ubuntu0.2 all
> [installed]
> 592,594c592,594
> < python3-twisted-bin/now 18.9.0-11ubuntu0.20.04.2 amd64
> [installed,upgradable to: 18.9.0-11ubuntu0.20.04.3]
> < python3-twisted/now 18.9.0-11ubuntu0.20.04.2 all [installed,upgradable
> to: 18.9.0-11ubuntu0.20.04.3]
> < python3-update-manager/now 1:20.04.10.11 all [installed,upgradable to:
> 1:20.04.10.20]
> ---
> > python3-twisted-bin/focal-updates,focal-security,now
> 18.9.0-11ubuntu0.20.04.3 amd64 [installed,automatic]
> > python3-twisted/focal-updates,focal-security,now
> 18.9.0-11ubuntu0.20.04.3 all [installed,automatic]
> > python3-update-manager/now 1:20.04.10.11 all [installed,upgradable to:
> 1:20.04.10.21]
> 618c618
> < snapd/now 2.58+20.04 amd64 [installed,upgradable to: 2.58+20.04.1]
> ---
> > snapd/now 2.58+20.04 amd64 [installed,upgradable to: 2.62+20.04]
> 625,627c625,627
> < sudo/now 1.8.31-1ubuntu1.4 amd64 [installed,upgradable to:
> 1.8.31-1ubuntu1.5]
> < systemd-sysv/now 245.4-4ubuntu3.19 amd64 [installed,upgradable to:
> 245.4-4ubuntu3.23]
> < systemd/now 245.4-4ubuntu3.19 amd64 [installed,upgradable to:
> 245.4-4ubuntu3.23]
> ---
> > sudo/focal-updates,focal-security,now 1.8.31-1ubuntu1.5 amd64
> [installed,automatic]
> > systemd-sysv/focal-security,now 245.4-4ubuntu3.20 amd64
> [installed,upgradable to: 245.4-4ubuntu3.23]
> > systemd/focal-security,now 245.4-4ubuntu3.20 amd64 [installed,upgradable
> to: 245.4-4ubuntu3.23]
> 637c637
> < ubuntu-advantage-tools/now 27.13.5~20.04.1 amd64 [installed,upgradable
> to: 31.2~20.04]
> ---
> > ubuntu-advantage-tools/now 27.13.5~20.04.1 amd64 [installed,upgradable
> to: 31.2.3~20.04]
> 648,649c648,649
> < update-manager-core/now 1:20.04.10.11 all [installed,upgradable to:
> 1:20.04.10.20]
> < update-notifier-common/now 3.192.30.16 all [installed,upgradable to:
> 3.192.30.17]
> ---
> > update-manager-core/now 1:20.04.10.11 all [installed,upgradable to:
> 1:20.04.10.21]
> > update-notifier-common/now 3.192.30.16 all [installed,upgradable to:
> 3.192.30.19]
> 654,655c654,655
> < util-linux/focal-security,now 2.34-0.1ubuntu9.3 amd64
> [installed,upgradable to: 2.34-0.1ubuntu9.4]
> < uuid-runtime/focal-security,now 2.34-0.1ubuntu9.3 amd64
> [installed,upgradable to: 2.34-0.1ubuntu9.4]
> ---
> > util-linux/now 2.34-0.1ubuntu9.3 amd64 [installed,upgradable to:
> 2.34-0.1ubuntu9.6]
> > uuid-runtime/now 2.34-0.1ubuntu9.3 amd64 [installed,upgradable to:
> 2.34-0.1ubuntu9.6]
> _______________________________________________
> varnish-misc mailing list
> varnish-misc at varnish-cache.org
> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/varnish-misc/attachments/20240518/e5c3ed9b/attachment-0001.html>
More information about the varnish-misc
mailing list