Varnish suddenly started using much more memory
Batanun B
batanun at hotmail.com
Fri May 17 13:15:10 UTC 2024
Hi,
Naturally, I can't be certain that the "in my mind" trivial VCL changes can't be the culprit. But I just can't see the logic in those changes causing this massive change in memory usage. But I'll summarize the changes here, and maybe you can identify a suspect:
* Modified the xkey header used by the xkey vmod, adding the id of the current website
* Modified the TTL, from 1w to 1h, for a specific type of resource existing in maybe 20 versions (ie different urls), each being about 5 kB in size
* Modified the backend probe url, from the startpage (ie full html) to a dedicated healthcheck endpoint (much smaller footprint, and much quicker)
That's it. That's all the VCL changes we made in that deployment. And, like I said, we did no changes in the OS or Varnish config.
> check the difference in passes, if they are about the same, look for hit-for-misses,
We don't have those statistics from the old server, so I can't do a comparison. But here are the current statistics:
MAIN.s_pass 180721 0.14 Total pass-ed requests seen
MAIN.cache_hitpass 0 0.00 Cache hits for pass.
MAIN.cache_hit 3718468 2.86 Cache hits
MAIN.cache_hit_grace 53903 0.04 Cache grace hits
MAIN.cache_hitpass 0 0.00 Cache hits for pass.
MAIN.cache_hitmiss 1129 0.00 Cache hits for miss.
> and lastly, look at how long Varnish is trying to cache the average object.
I'm not sure how I do that. Is there a varnishstat counter I can look at?
> which packages are you using?
Instead of giving you the full list, I guess it makes more sense to just list the one that differ. Below is a diff output of "apt list --installed" of before and after the deploy.
Regards
2c2
< accountsservice/now 0.6.55-0ubuntu12~20.04.5 amd64 [installed,upgradable to: 0.6.55-0ubuntu12~20.04.7]
---
> accountsservice/focal-updates,focal-security,now 0.6.55-0ubuntu12~20.04.7 amd64 [installed,automatic]
23,25c23,25
< bind9-dnsutils/now 1:9.16.1-0ubuntu2.12 amd64 [installed,upgradable to: 1:9.16.48-0ubuntu0.20.04.1]
< bind9-host/now 1:9.16.1-0ubuntu2.12 amd64 [installed,upgradable to: 1:9.16.48-0ubuntu0.20.04.1]
< bind9-libs/now 1:9.16.1-0ubuntu2.12 amd64 [installed,upgradable to: 1:9.16.48-0ubuntu0.20.04.1]
---
> bind9-dnsutils/focal-updates,focal-security,now 1:9.16.48-0ubuntu0.20.04.1 amd64 [installed,automatic]
> bind9-host/focal-updates,focal-security,now 1:9.16.48-0ubuntu0.20.04.1 amd64 [installed,automatic]
> bind9-libs/focal-updates,focal-security,now 1:9.16.48-0ubuntu0.20.04.1 amd64 [installed,automatic]
31c31
< bsdutils/focal-security,now 1:2.34-0.1ubuntu9.3 amd64 [installed,upgradable to: 1:2.34-0.1ubuntu9.4]
---
> bsdutils/now 1:2.34-0.1ubuntu9.3 amd64 [installed,upgradable to: 1:2.34-0.1ubuntu9.6]
41c41
< cloud-init/now 22.4.2-0ubuntu0~20.04.2 all [installed,upgradable to: 23.4.4-0ubuntu0~20.04.1]
---
> cloud-init/now 22.4.2-0ubuntu0~20.04.2 all [installed,upgradable to: 24.1.3-0ubuntu1~20.04.1]
48c48
< cpio/focal-updates,focal-security,now 2.13+dfsg-2ubuntu0.3 amd64 [installed,automatic]
---
> cpio/now 2.13+dfsg-2ubuntu0.3 amd64 [installed,upgradable to: 2.13+dfsg-2ubuntu0.4]
56c56
< curl/focal-updates,focal-security,now 7.68.0-1ubuntu2.21 amd64 [installed]
---
> curl/focal-updates,focal-security,now 7.68.0-1ubuntu2.22 amd64 [installed]
68c68
< distro-info-data/now 0.43ubuntu1.11 all [installed,upgradable to: 0.43ubuntu1.15]
---
> distro-info-data/now 0.43ubuntu1.11 all [installed,upgradable to: 0.43ubuntu1.16]
82c82
< fdisk/focal-security,now 2.34-0.1ubuntu9.3 amd64 [installed,upgradable to: 2.34-0.1ubuntu9.4]
---
> fdisk/now 2.34-0.1ubuntu9.3 amd64 [installed,upgradable to: 2.34-0.1ubuntu9.6]
119,120c119,120
< grub-efi-amd64-bin/now 2.06-2ubuntu14.1 amd64 [installed,upgradable to: 2.06-2ubuntu14.4]
< grub-efi-amd64-signed/now 1.187.3~20.04.1+2.06-2ubuntu14.1 amd64 [installed,upgradable to: 1.187.6~20.04.1+2.06-2ubuntu14.4]
---
> grub-efi-amd64-bin/focal-updates,focal-security,now 2.06-2ubuntu14.4 amd64 [installed]
> grub-efi-amd64-signed/focal-updates,focal-security,now 1.187.6~20.04.1+2.06-2ubuntu14.4 amd64 [installed]
148c148
< klibc-utils/focal-updates,focal-security,now 2.0.7-1ubuntu5.1 amd64 [installed,automatic]
---
> klibc-utils/focal-updates,focal-security,now 2.0.7-1ubuntu5.2 amd64 [installed,automatic]
152c152
< landscape-common/focal-updates,now 19.12-0ubuntu4.3 amd64 [installed,automatic]
---
> landscape-common/now 19.12-0ubuntu4.3 amd64 [installed,upgradable to: 23.02-0ubuntu1~20.04.2]
154,155c154,155
< less/now 551-1ubuntu0.1 amd64 [installed,upgradable to: 551-1ubuntu0.2]
< libaccountsservice0/now 0.6.55-0ubuntu12~20.04.5 amd64 [installed,upgradable to: 0.6.55-0ubuntu12~20.04.7]
---
> less/now 551-1ubuntu0.1 amd64 [installed,upgradable to: 551-1ubuntu0.3]
> libaccountsservice0/focal-updates,focal-security,now 0.6.55-0ubuntu12~20.04.7 amd64 [installed,automatic]
175c175
< libblkid1/focal-security,now 2.34-0.1ubuntu9.3 amd64 [installed,upgradable to: 2.34-0.1ubuntu9.4]
---
> libblkid1/now 2.34-0.1ubuntu9.3 amd64 [installed,upgradable to: 2.34-0.1ubuntu9.6]
187,190c187,190
< libc-bin/now 2.31-0ubuntu9.9 amd64 [installed,upgradable to: 2.31-0ubuntu9.14]
< libc-dev-bin/focal-updates,focal-security,now 2.31-0ubuntu9.14 amd64 [installed,automatic]
< libc6-dev/focal-updates,focal-security,now 2.31-0ubuntu9.14 amd64 [installed,automatic]
< libc6/focal-updates,focal-security,now 2.31-0ubuntu9.14 amd64 [installed,automatic]
---
> libc-bin/now 2.31-0ubuntu9.9 amd64 [installed,upgradable to: 2.31-0ubuntu9.15]
> libc-dev-bin/focal-updates,focal-security,now 2.31-0ubuntu9.15 amd64 [installed,automatic]
> libc6-dev/focal-updates,focal-security,now 2.31-0ubuntu9.15 amd64 [installed,automatic]
> libc6/focal-updates,focal-security,now 2.31-0ubuntu9.15 amd64 [installed,automatic]
203,204c203,204
< libcurl3-gnutls/now 7.68.0-1ubuntu2.15 amd64 [installed,upgradable to: 7.68.0-1ubuntu2.21]
< libcurl4/focal-updates,focal-security,now 7.68.0-1ubuntu2.21 amd64 [installed,automatic]
---
> libcurl3-gnutls/now 7.68.0-1ubuntu2.15 amd64 [installed,upgradable to: 7.68.0-1ubuntu2.22]
> libcurl4/focal-updates,focal-security,now 7.68.0-1ubuntu2.22 amd64 [installed,automatic]
227c227
< libfdisk1/focal-security,now 2.34-0.1ubuntu9.3 amd64 [installed,upgradable to: 2.34-0.1ubuntu9.4]
---
> libfdisk1/now 2.34-0.1ubuntu9.3 amd64 [installed,upgradable to: 2.34-0.1ubuntu9.6]
244,246c244,246
< libglib2.0-0/now 2.64.6-1~ubuntu20.04.4 amd64 [installed,upgradable to: 2.64.6-1~ubuntu20.04.6]
< libglib2.0-bin/now 2.64.6-1~ubuntu20.04.4 amd64 [installed,upgradable to: 2.64.6-1~ubuntu20.04.6]
< libglib2.0-data/now 2.64.6-1~ubuntu20.04.4 all [installed,upgradable to: 2.64.6-1~ubuntu20.04.6]
---
> libglib2.0-0/now 2.64.6-1~ubuntu20.04.4 amd64 [installed,upgradable to: 2.64.6-1~ubuntu20.04.7]
> libglib2.0-bin/now 2.64.6-1~ubuntu20.04.4 amd64 [installed,upgradable to: 2.64.6-1~ubuntu20.04.7]
> libglib2.0-data/now 2.64.6-1~ubuntu20.04.4 all [installed,upgradable to: 2.64.6-1~ubuntu20.04.7]
248c248
< libgnutls30/now 3.6.13-2ubuntu1.7 amd64 [installed,upgradable to: 3.6.13-2ubuntu1.10]
---
> libgnutls30/now 3.6.13-2ubuntu1.7 amd64 [installed,upgradable to: 3.6.13-2ubuntu1.11]
278c278
< libklibc/focal-updates,focal-security,now 2.0.7-1ubuntu5.1 amd64 [installed,automatic]
---
> libklibc/focal-updates,focal-security,now 2.0.7-1ubuntu5.2 amd64 [installed,automatic]
284c284
< libldap-2.4-2/now 2.4.49+dfsg-2ubuntu1.9 amd64 [installed,upgradable to: 2.4.49+dfsg-2ubuntu1.10]
---
> libldap-2.4-2/focal-updates,focal-security,now 2.4.49+dfsg-2ubuntu1.10 amd64 [installed,automatic]
301c301
< libmount1/focal-security,now 2.34-0.1ubuntu9.3 amd64 [installed,upgradable to: 2.34-0.1ubuntu9.4]
---
> libmount1/now 2.34-0.1ubuntu9.3 amd64 [installed,upgradable to: 2.34-0.1ubuntu9.6]
306,307c306,307
< libncurses6/focal,now 6.2-0ubuntu2 amd64 [installed,upgradable to: 6.2-0ubuntu2.1]
< libncursesw6/focal,now 6.2-0ubuntu2 amd64 [installed,upgradable to: 6.2-0ubuntu2.1]
---
> libncurses6/focal-updates,focal-security,now 6.2-0ubuntu2.1 amd64 [installed,automatic]
> libncursesw6/focal-updates,focal-security,now 6.2-0ubuntu2.1 amd64 [installed,automatic]
314c314
< libnghttp2-14/focal,now 1.40.0-1build1 amd64 [installed,upgradable to: 1.40.0-1ubuntu0.2]
---
> libnghttp2-14/focal,now 1.40.0-1build1 amd64 [installed,upgradable to: 1.40.0-1ubuntu0.3]
316,318c316,318
< libnspr4/focal,now 2:4.25-1 amd64 [installed,automatic]
< libnss-systemd/now 245.4-4ubuntu3.19 amd64 [installed,upgradable to: 245.4-4ubuntu3.23]
< libnss3/now 2:3.49.1-1ubuntu1.8 amd64 [installed,upgradable to: 2:3.49.1-1ubuntu1.9]
---
> libnspr4/focal,now 2:4.25-1 amd64 [installed,upgradable to: 2:4.35-0ubuntu0.20.04.1]
> libnss-systemd/focal-security,now 245.4-4ubuntu3.20 amd64 [installed,upgradable to: 245.4-4ubuntu3.23]
> libnss3/now 2:3.49.1-1ubuntu1.8 amd64 [installed,upgradable to: 2:3.98-0ubuntu0.20.04.2]
325,328c325,328
< libpam-modules-bin/now 1.3.1-5ubuntu4.6 amd64 [installed,upgradable to: 1.3.1-5ubuntu4.7]
< libpam-modules/now 1.3.1-5ubuntu4.6 amd64 [installed,upgradable to: 1.3.1-5ubuntu4.7]
< libpam-runtime/now 1.3.1-5ubuntu4.6 all [installed,upgradable to: 1.3.1-5ubuntu4.7]
< libpam-systemd/now 245.4-4ubuntu3.19 amd64 [installed,upgradable to: 245.4-4ubuntu3.23]
---
> libpam-modules-bin/focal-updates,focal-security,now 1.3.1-5ubuntu4.7 amd64 [installed,automatic]
> libpam-modules/focal-updates,focal-security,now 1.3.1-5ubuntu4.7 amd64 [installed,automatic]
> libpam-runtime/focal-updates,focal-security,now 1.3.1-5ubuntu4.7 all [installed,automatic]
> libpam-systemd/focal-security,now 245.4-4ubuntu3.20 amd64 [installed,upgradable to: 245.4-4ubuntu3.23]
347c347
< libprocps8/now 2:3.3.16-1ubuntu2.3 amd64 [installed,upgradable to: 2:3.3.16-1ubuntu2.4]
---
> libprocps8/focal-updates,focal-security,now 2:3.3.16-1ubuntu2.4 amd64 [installed,automatic]
375c375
< libsmartcols1/focal-security,now 2.34-0.1ubuntu9.3 amd64 [installed,upgradable to: 2.34-0.1ubuntu9.4]
---
> libsmartcols1/now 2.34-0.1ubuntu9.3 amd64 [installed,upgradable to: 2.34-0.1ubuntu9.6]
386c386
< libsystemd0/now 245.4-4ubuntu3.19 amd64 [installed,upgradable to: 245.4-4ubuntu3.23]
---
> libsystemd0/focal-security,now 245.4-4ubuntu3.20 amd64 [installed,upgradable to: 245.4-4ubuntu3.23]
394c394
< libtinfo6/focal,now 6.2-0ubuntu2 amd64 [installed,upgradable to: 6.2-0ubuntu2.1]
---
> libtinfo6/focal-updates,focal-security,now 6.2-0ubuntu2.1 amd64 [installed,automatic]
407c407
< libuuid1/focal-security,now 2.34-0.1ubuntu9.3 amd64 [installed,upgradable to: 2.34-0.1ubuntu9.4]
---
> libuuid1/now 2.34-0.1ubuntu9.3 amd64 [installed,upgradable to: 2.34-0.1ubuntu9.6]
434c434
< linux-azure/now 5.15.0.1033.40~20.04.23 amd64 [installed,upgradable to: 5.15.0.1059.67~20.04.49]
---
> linux-azure/now 5.15.0.1033.40~20.04.23 amd64 [installed,upgradable to: 5.15.0.1064.73~20.04.1]
438,439c438,439
< linux-cloud-tools-azure/now 5.15.0.1033.40~20.04.23 amd64 [installed,upgradable to: 5.15.0.1059.67~20.04.49]
< linux-cloud-tools-common/focal-updates,focal-security,now 5.4.0-139.156 all [installed,upgradable to: 5.4.0-174.193]
---
> linux-cloud-tools-azure/now 5.15.0.1033.40~20.04.23 amd64 [installed,upgradable to: 5.15.0.1064.73~20.04.1]
> linux-cloud-tools-common/focal-updates,focal-security,now 5.4.0-182.202 all [installed]
441c441
< linux-headers-azure/now 5.15.0.1033.40~20.04.23 amd64 [installed,upgradable to: 5.15.0.1059.67~20.04.49]
---
> linux-headers-azure/now 5.15.0.1033.40~20.04.23 amd64 [installed,upgradable to: 5.15.0.1064.73~20.04.1]
443,444c443,444
< linux-image-azure/now 5.15.0.1033.40~20.04.23 amd64 [installed,upgradable to: 5.15.0.1059.67~20.04.49]
< linux-libc-dev/focal-updates,focal-security,now 5.4.0-174.193 amd64 [installed,automatic]
---
> linux-image-azure/now 5.15.0.1033.40~20.04.23 amd64 [installed,upgradable to: 5.15.0.1064.73~20.04.1]
> linux-libc-dev/focal-updates,focal-security,now 5.4.0-182.202 amd64 [installed,automatic]
447,449c447,449
< linux-tools-azure/now 5.15.0.1033.40~20.04.23 amd64 [installed,upgradable to: 5.15.0.1059.67~20.04.49]
< linux-tools-common/focal-updates,focal-security,now 5.4.0-139.156 all [installed,upgradable to: 5.4.0-174.193]
< locales/now 2.31-0ubuntu9.9 all [installed,upgradable to: 2.31-0ubuntu9.14]
---
> linux-tools-azure/now 5.15.0.1033.40~20.04.23 amd64 [installed,upgradable to: 5.15.0.1064.73~20.04.1]
> linux-tools-common/focal-updates,focal-security,now 5.4.0-139.156 all [installed,upgradable to: 5.4.0-182.202]
> locales/now 2.31-0ubuntu9.9 all [installed,upgradable to: 2.31-0ubuntu9.15]
473c473
< mount/focal-security,now 2.34-0.1ubuntu9.3 amd64 [installed,upgradable to: 2.34-0.1ubuntu9.4]
---
> mount/focal-updates,focal-security,now 2.34-0.1ubuntu9.6 amd64 [installed,automatic]
499c499
< passwd/now 1:4.8.1-1ubuntu5.20.04.4 amd64 [installed,upgradable to: 1:4.8.1-1ubuntu5.20.04.5]
---
> passwd/focal-updates,focal-security,now 1:4.8.1-1ubuntu5.20.04.5 amd64 [installed,automatic]
555c555
< python3-jinja2/focal,now 2.10.1-2 all [installed,upgradable to: 2.10.1-2ubuntu0.2]
---
> python3-jinja2/focal-updates,focal-security,now 2.10.1-2ubuntu0.2 all [installed]
592,594c592,594
< python3-twisted-bin/now 18.9.0-11ubuntu0.20.04.2 amd64 [installed,upgradable to: 18.9.0-11ubuntu0.20.04.3]
< python3-twisted/now 18.9.0-11ubuntu0.20.04.2 all [installed,upgradable to: 18.9.0-11ubuntu0.20.04.3]
< python3-update-manager/now 1:20.04.10.11 all [installed,upgradable to: 1:20.04.10.20]
---
> python3-twisted-bin/focal-updates,focal-security,now 18.9.0-11ubuntu0.20.04.3 amd64 [installed,automatic]
> python3-twisted/focal-updates,focal-security,now 18.9.0-11ubuntu0.20.04.3 all [installed,automatic]
> python3-update-manager/now 1:20.04.10.11 all [installed,upgradable to: 1:20.04.10.21]
618c618
< snapd/now 2.58+20.04 amd64 [installed,upgradable to: 2.58+20.04.1]
---
> snapd/now 2.58+20.04 amd64 [installed,upgradable to: 2.62+20.04]
625,627c625,627
< sudo/now 1.8.31-1ubuntu1.4 amd64 [installed,upgradable to: 1.8.31-1ubuntu1.5]
< systemd-sysv/now 245.4-4ubuntu3.19 amd64 [installed,upgradable to: 245.4-4ubuntu3.23]
< systemd/now 245.4-4ubuntu3.19 amd64 [installed,upgradable to: 245.4-4ubuntu3.23]
---
> sudo/focal-updates,focal-security,now 1.8.31-1ubuntu1.5 amd64 [installed,automatic]
> systemd-sysv/focal-security,now 245.4-4ubuntu3.20 amd64 [installed,upgradable to: 245.4-4ubuntu3.23]
> systemd/focal-security,now 245.4-4ubuntu3.20 amd64 [installed,upgradable to: 245.4-4ubuntu3.23]
637c637
< ubuntu-advantage-tools/now 27.13.5~20.04.1 amd64 [installed,upgradable to: 31.2~20.04]
---
> ubuntu-advantage-tools/now 27.13.5~20.04.1 amd64 [installed,upgradable to: 31.2.3~20.04]
648,649c648,649
< update-manager-core/now 1:20.04.10.11 all [installed,upgradable to: 1:20.04.10.20]
< update-notifier-common/now 3.192.30.16 all [installed,upgradable to: 3.192.30.17]
---
> update-manager-core/now 1:20.04.10.11 all [installed,upgradable to: 1:20.04.10.21]
> update-notifier-common/now 3.192.30.16 all [installed,upgradable to: 3.192.30.19]
654,655c654,655
< util-linux/focal-security,now 2.34-0.1ubuntu9.3 amd64 [installed,upgradable to: 2.34-0.1ubuntu9.4]
< uuid-runtime/focal-security,now 2.34-0.1ubuntu9.3 amd64 [installed,upgradable to: 2.34-0.1ubuntu9.4]
---
> util-linux/now 2.34-0.1ubuntu9.3 amd64 [installed,upgradable to: 2.34-0.1ubuntu9.6]
> uuid-runtime/now 2.34-0.1ubuntu9.3 amd64 [installed,upgradable to: 2.34-0.1ubuntu9.6]
More information about the varnish-misc
mailing list