<div dir="ltr"><div dir="auto">Sorry, I should have been clearer, I meant: where are the varnish packages coming from? Are they from the official repositories, from <a href="https://packagecloud.io/varnishcache/">https://packagecloud.io/varnishcache/</a> or built from source maybe?</div><div dir="auto"><br></div><div>If you don't have old metrics (you should really invest some time in something like prometheus, it would probably have made the issue obvious), then we can't really compare anything. Is there any chance you can run the old version on the server to explore the differences?</div><div><br></div><div>Two extra questions:</div><div>- what's the output of: varnishstat -1 -f '*g_bytes'</div><div>- have you tweaked any workspaces/thread parameters?</div><div><br></div><div>Cheers,<br></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Fri, May 17, 2024, 06:17 Batanun B <<a href="mailto:batanun@hotmail.com" target="_blank">batanun@hotmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">Hi,<br>
<br>
Naturally, I can't be certain that the "in my mind" trivial VCL changes can't be the culprit. But I just can't see the logic in those changes causing this massive change in memory usage. But I'll summarize the changes here, and maybe you can identify a suspect:<br>
<br>
* Modified the xkey header used by the xkey vmod, adding the id of the current website<br>
* Modified the TTL, from 1w to 1h, for a specific type of resource existing in maybe 20 versions (ie different urls), each being about 5 kB in size<br>
* Modified the backend probe url, from the startpage (ie full html) to a dedicated healthcheck endpoint (much smaller footprint, and much quicker)<br>
<br>
That's it. That's all the VCL changes we made in that deployment. And, like I said, we did no changes in the OS or Varnish config.<br>
<br>
<br>
> check the difference in passes, if they are about the same, look for hit-for-misses,<br>
<br>
We don't have those statistics from the old server, so I can't do a comparison. But here are the current statistics:<br>
<br>
MAIN.s_pass 180721 0.14 Total pass-ed requests seen<br>
MAIN.cache_hitpass 0 0.00 Cache hits for pass.<br>
MAIN.cache_hit 3718468 2.86 Cache hits<br>
MAIN.cache_hit_grace 53903 0.04 Cache grace hits<br>
MAIN.cache_hitpass 0 0.00 Cache hits for pass.<br>
MAIN.cache_hitmiss 1129 0.00 Cache hits for miss.<br>
<br>
<br>
> and lastly, look at how long Varnish is trying to cache the average object.<br>
<br>
I'm not sure how I do that. Is there a varnishstat counter I can look at?<br>
<br>
> which packages are you using?<br>
<br>
Instead of giving you the full list, I guess it makes more sense to just list the one that differ. Below is a diff output of "apt list --installed" of before and after the deploy.<br>
<br>
Regards<br>
<br>
<br>
2c2<br>
< accountsservice/now 0.6.55-0ubuntu12~20.04.5 amd64 [installed,upgradable to: 0.6.55-0ubuntu12~20.04.7]<br>
---<br>
> accountsservice/focal-updates,focal-security,now 0.6.55-0ubuntu12~20.04.7 amd64 [installed,automatic]<br>
23,25c23,25<br>
< bind9-dnsutils/now 1:9.16.1-0ubuntu2.12 amd64 [installed,upgradable to: 1:9.16.48-0ubuntu0.20.04.1]<br>
< bind9-host/now 1:9.16.1-0ubuntu2.12 amd64 [installed,upgradable to: 1:9.16.48-0ubuntu0.20.04.1]<br>
< bind9-libs/now 1:9.16.1-0ubuntu2.12 amd64 [installed,upgradable to: 1:9.16.48-0ubuntu0.20.04.1]<br>
---<br>
> bind9-dnsutils/focal-updates,focal-security,now 1:9.16.48-0ubuntu0.20.04.1 amd64 [installed,automatic]<br>
> bind9-host/focal-updates,focal-security,now 1:9.16.48-0ubuntu0.20.04.1 amd64 [installed,automatic]<br>
> bind9-libs/focal-updates,focal-security,now 1:9.16.48-0ubuntu0.20.04.1 amd64 [installed,automatic]<br>
31c31<br>
< bsdutils/focal-security,now 1:2.34-0.1ubuntu9.3 amd64 [installed,upgradable to: 1:2.34-0.1ubuntu9.4]<br>
---<br>
> bsdutils/now 1:2.34-0.1ubuntu9.3 amd64 [installed,upgradable to: 1:2.34-0.1ubuntu9.6]<br>
41c41<br>
< cloud-init/now 22.4.2-0ubuntu0~20.04.2 all [installed,upgradable to: 23.4.4-0ubuntu0~20.04.1]<br>
---<br>
> cloud-init/now 22.4.2-0ubuntu0~20.04.2 all [installed,upgradable to: 24.1.3-0ubuntu1~20.04.1]<br>
48c48<br>
< cpio/focal-updates,focal-security,now 2.13+dfsg-2ubuntu0.3 amd64 [installed,automatic]<br>
---<br>
> cpio/now 2.13+dfsg-2ubuntu0.3 amd64 [installed,upgradable to: 2.13+dfsg-2ubuntu0.4]<br>
56c56<br>
< curl/focal-updates,focal-security,now 7.68.0-1ubuntu2.21 amd64 [installed]<br>
---<br>
> curl/focal-updates,focal-security,now 7.68.0-1ubuntu2.22 amd64 [installed]<br>
68c68<br>
< distro-info-data/now 0.43ubuntu1.11 all [installed,upgradable to: 0.43ubuntu1.15]<br>
---<br>
> distro-info-data/now 0.43ubuntu1.11 all [installed,upgradable to: 0.43ubuntu1.16]<br>
82c82<br>
< fdisk/focal-security,now 2.34-0.1ubuntu9.3 amd64 [installed,upgradable to: 2.34-0.1ubuntu9.4]<br>
---<br>
> fdisk/now 2.34-0.1ubuntu9.3 amd64 [installed,upgradable to: 2.34-0.1ubuntu9.6]<br>
119,120c119,120<br>
< grub-efi-amd64-bin/now 2.06-2ubuntu14.1 amd64 [installed,upgradable to: 2.06-2ubuntu14.4]<br>
< grub-efi-amd64-signed/now 1.187.3~20.04.1+2.06-2ubuntu14.1 amd64 [installed,upgradable to: 1.187.6~20.04.1+2.06-2ubuntu14.4]<br>
---<br>
> grub-efi-amd64-bin/focal-updates,focal-security,now 2.06-2ubuntu14.4 amd64 [installed]<br>
> grub-efi-amd64-signed/focal-updates,focal-security,now 1.187.6~20.04.1+2.06-2ubuntu14.4 amd64 [installed]<br>
148c148<br>
< klibc-utils/focal-updates,focal-security,now 2.0.7-1ubuntu5.1 amd64 [installed,automatic]<br>
---<br>
> klibc-utils/focal-updates,focal-security,now 2.0.7-1ubuntu5.2 amd64 [installed,automatic]<br>
152c152<br>
< landscape-common/focal-updates,now 19.12-0ubuntu4.3 amd64 [installed,automatic]<br>
---<br>
> landscape-common/now 19.12-0ubuntu4.3 amd64 [installed,upgradable to: 23.02-0ubuntu1~20.04.2]<br>
154,155c154,155<br>
< less/now 551-1ubuntu0.1 amd64 [installed,upgradable to: 551-1ubuntu0.2]<br>
< libaccountsservice0/now 0.6.55-0ubuntu12~20.04.5 amd64 [installed,upgradable to: 0.6.55-0ubuntu12~20.04.7]<br>
---<br>
> less/now 551-1ubuntu0.1 amd64 [installed,upgradable to: 551-1ubuntu0.3]<br>
> libaccountsservice0/focal-updates,focal-security,now 0.6.55-0ubuntu12~20.04.7 amd64 [installed,automatic]<br>
175c175<br>
< libblkid1/focal-security,now 2.34-0.1ubuntu9.3 amd64 [installed,upgradable to: 2.34-0.1ubuntu9.4]<br>
---<br>
> libblkid1/now 2.34-0.1ubuntu9.3 amd64 [installed,upgradable to: 2.34-0.1ubuntu9.6]<br>
187,190c187,190<br>
< libc-bin/now 2.31-0ubuntu9.9 amd64 [installed,upgradable to: 2.31-0ubuntu9.14]<br>
< libc-dev-bin/focal-updates,focal-security,now 2.31-0ubuntu9.14 amd64 [installed,automatic]<br>
< libc6-dev/focal-updates,focal-security,now 2.31-0ubuntu9.14 amd64 [installed,automatic]<br>
< libc6/focal-updates,focal-security,now 2.31-0ubuntu9.14 amd64 [installed,automatic]<br>
---<br>
> libc-bin/now 2.31-0ubuntu9.9 amd64 [installed,upgradable to: 2.31-0ubuntu9.15]<br>
> libc-dev-bin/focal-updates,focal-security,now 2.31-0ubuntu9.15 amd64 [installed,automatic]<br>
> libc6-dev/focal-updates,focal-security,now 2.31-0ubuntu9.15 amd64 [installed,automatic]<br>
> libc6/focal-updates,focal-security,now 2.31-0ubuntu9.15 amd64 [installed,automatic]<br>
203,204c203,204<br>
< libcurl3-gnutls/now 7.68.0-1ubuntu2.15 amd64 [installed,upgradable to: 7.68.0-1ubuntu2.21]<br>
< libcurl4/focal-updates,focal-security,now 7.68.0-1ubuntu2.21 amd64 [installed,automatic]<br>
---<br>
> libcurl3-gnutls/now 7.68.0-1ubuntu2.15 amd64 [installed,upgradable to: 7.68.0-1ubuntu2.22]<br>
> libcurl4/focal-updates,focal-security,now 7.68.0-1ubuntu2.22 amd64 [installed,automatic]<br>
227c227<br>
< libfdisk1/focal-security,now 2.34-0.1ubuntu9.3 amd64 [installed,upgradable to: 2.34-0.1ubuntu9.4]<br>
---<br>
> libfdisk1/now 2.34-0.1ubuntu9.3 amd64 [installed,upgradable to: 2.34-0.1ubuntu9.6]<br>
244,246c244,246<br>
< libglib2.0-0/now 2.64.6-1~ubuntu20.04.4 amd64 [installed,upgradable to: 2.64.6-1~ubuntu20.04.6]<br>
< libglib2.0-bin/now 2.64.6-1~ubuntu20.04.4 amd64 [installed,upgradable to: 2.64.6-1~ubuntu20.04.6]<br>
< libglib2.0-data/now 2.64.6-1~ubuntu20.04.4 all [installed,upgradable to: 2.64.6-1~ubuntu20.04.6]<br>
---<br>
> libglib2.0-0/now 2.64.6-1~ubuntu20.04.4 amd64 [installed,upgradable to: 2.64.6-1~ubuntu20.04.7]<br>
> libglib2.0-bin/now 2.64.6-1~ubuntu20.04.4 amd64 [installed,upgradable to: 2.64.6-1~ubuntu20.04.7]<br>
> libglib2.0-data/now 2.64.6-1~ubuntu20.04.4 all [installed,upgradable to: 2.64.6-1~ubuntu20.04.7]<br>
248c248<br>
< libgnutls30/now 3.6.13-2ubuntu1.7 amd64 [installed,upgradable to: 3.6.13-2ubuntu1.10]<br>
---<br>
> libgnutls30/now 3.6.13-2ubuntu1.7 amd64 [installed,upgradable to: 3.6.13-2ubuntu1.11]<br>
278c278<br>
< libklibc/focal-updates,focal-security,now 2.0.7-1ubuntu5.1 amd64 [installed,automatic]<br>
---<br>
> libklibc/focal-updates,focal-security,now 2.0.7-1ubuntu5.2 amd64 [installed,automatic]<br>
284c284<br>
< libldap-2.4-2/now 2.4.49+dfsg-2ubuntu1.9 amd64 [installed,upgradable to: 2.4.49+dfsg-2ubuntu1.10]<br>
---<br>
> libldap-2.4-2/focal-updates,focal-security,now 2.4.49+dfsg-2ubuntu1.10 amd64 [installed,automatic]<br>
301c301<br>
< libmount1/focal-security,now 2.34-0.1ubuntu9.3 amd64 [installed,upgradable to: 2.34-0.1ubuntu9.4]<br>
---<br>
> libmount1/now 2.34-0.1ubuntu9.3 amd64 [installed,upgradable to: 2.34-0.1ubuntu9.6]<br>
306,307c306,307<br>
< libncurses6/focal,now 6.2-0ubuntu2 amd64 [installed,upgradable to: 6.2-0ubuntu2.1]<br>
< libncursesw6/focal,now 6.2-0ubuntu2 amd64 [installed,upgradable to: 6.2-0ubuntu2.1]<br>
---<br>
> libncurses6/focal-updates,focal-security,now 6.2-0ubuntu2.1 amd64 [installed,automatic]<br>
> libncursesw6/focal-updates,focal-security,now 6.2-0ubuntu2.1 amd64 [installed,automatic]<br>
314c314<br>
< libnghttp2-14/focal,now 1.40.0-1build1 amd64 [installed,upgradable to: 1.40.0-1ubuntu0.2]<br>
---<br>
> libnghttp2-14/focal,now 1.40.0-1build1 amd64 [installed,upgradable to: 1.40.0-1ubuntu0.3]<br>
316,318c316,318<br>
< libnspr4/focal,now 2:4.25-1 amd64 [installed,automatic]<br>
< libnss-systemd/now 245.4-4ubuntu3.19 amd64 [installed,upgradable to: 245.4-4ubuntu3.23]<br>
< libnss3/now 2:3.49.1-1ubuntu1.8 amd64 [installed,upgradable to: 2:3.49.1-1ubuntu1.9]<br>
---<br>
> libnspr4/focal,now 2:4.25-1 amd64 [installed,upgradable to: 2:4.35-0ubuntu0.20.04.1]<br>
> libnss-systemd/focal-security,now 245.4-4ubuntu3.20 amd64 [installed,upgradable to: 245.4-4ubuntu3.23]<br>
> libnss3/now 2:3.49.1-1ubuntu1.8 amd64 [installed,upgradable to: 2:3.98-0ubuntu0.20.04.2]<br>
325,328c325,328<br>
< libpam-modules-bin/now 1.3.1-5ubuntu4.6 amd64 [installed,upgradable to: 1.3.1-5ubuntu4.7]<br>
< libpam-modules/now 1.3.1-5ubuntu4.6 amd64 [installed,upgradable to: 1.3.1-5ubuntu4.7]<br>
< libpam-runtime/now 1.3.1-5ubuntu4.6 all [installed,upgradable to: 1.3.1-5ubuntu4.7]<br>
< libpam-systemd/now 245.4-4ubuntu3.19 amd64 [installed,upgradable to: 245.4-4ubuntu3.23]<br>
---<br>
> libpam-modules-bin/focal-updates,focal-security,now 1.3.1-5ubuntu4.7 amd64 [installed,automatic]<br>
> libpam-modules/focal-updates,focal-security,now 1.3.1-5ubuntu4.7 amd64 [installed,automatic]<br>
> libpam-runtime/focal-updates,focal-security,now 1.3.1-5ubuntu4.7 all [installed,automatic]<br>
> libpam-systemd/focal-security,now 245.4-4ubuntu3.20 amd64 [installed,upgradable to: 245.4-4ubuntu3.23]<br>
347c347<br>
< libprocps8/now 2:3.3.16-1ubuntu2.3 amd64 [installed,upgradable to: 2:3.3.16-1ubuntu2.4]<br>
---<br>
> libprocps8/focal-updates,focal-security,now 2:3.3.16-1ubuntu2.4 amd64 [installed,automatic]<br>
375c375<br>
< libsmartcols1/focal-security,now 2.34-0.1ubuntu9.3 amd64 [installed,upgradable to: 2.34-0.1ubuntu9.4]<br>
---<br>
> libsmartcols1/now 2.34-0.1ubuntu9.3 amd64 [installed,upgradable to: 2.34-0.1ubuntu9.6]<br>
386c386<br>
< libsystemd0/now 245.4-4ubuntu3.19 amd64 [installed,upgradable to: 245.4-4ubuntu3.23]<br>
---<br>
> libsystemd0/focal-security,now 245.4-4ubuntu3.20 amd64 [installed,upgradable to: 245.4-4ubuntu3.23]<br>
394c394<br>
< libtinfo6/focal,now 6.2-0ubuntu2 amd64 [installed,upgradable to: 6.2-0ubuntu2.1]<br>
---<br>
> libtinfo6/focal-updates,focal-security,now 6.2-0ubuntu2.1 amd64 [installed,automatic]<br>
407c407<br>
< libuuid1/focal-security,now 2.34-0.1ubuntu9.3 amd64 [installed,upgradable to: 2.34-0.1ubuntu9.4]<br>
---<br>
> libuuid1/now 2.34-0.1ubuntu9.3 amd64 [installed,upgradable to: 2.34-0.1ubuntu9.6]<br>
434c434<br>
< linux-azure/now 5.15.0.1033.40~20.04.23 amd64 [installed,upgradable to: 5.15.0.1059.67~20.04.49]<br>
---<br>
> linux-azure/now 5.15.0.1033.40~20.04.23 amd64 [installed,upgradable to: 5.15.0.1064.73~20.04.1]<br>
438,439c438,439<br>
< linux-cloud-tools-azure/now 5.15.0.1033.40~20.04.23 amd64 [installed,upgradable to: 5.15.0.1059.67~20.04.49]<br>
< linux-cloud-tools-common/focal-updates,focal-security,now 5.4.0-139.156 all [installed,upgradable to: 5.4.0-174.193]<br>
---<br>
> linux-cloud-tools-azure/now 5.15.0.1033.40~20.04.23 amd64 [installed,upgradable to: 5.15.0.1064.73~20.04.1]<br>
> linux-cloud-tools-common/focal-updates,focal-security,now 5.4.0-182.202 all [installed]<br>
441c441<br>
< linux-headers-azure/now 5.15.0.1033.40~20.04.23 amd64 [installed,upgradable to: 5.15.0.1059.67~20.04.49]<br>
---<br>
> linux-headers-azure/now 5.15.0.1033.40~20.04.23 amd64 [installed,upgradable to: 5.15.0.1064.73~20.04.1]<br>
443,444c443,444<br>
< linux-image-azure/now 5.15.0.1033.40~20.04.23 amd64 [installed,upgradable to: 5.15.0.1059.67~20.04.49]<br>
< linux-libc-dev/focal-updates,focal-security,now 5.4.0-174.193 amd64 [installed,automatic]<br>
---<br>
> linux-image-azure/now 5.15.0.1033.40~20.04.23 amd64 [installed,upgradable to: 5.15.0.1064.73~20.04.1]<br>
> linux-libc-dev/focal-updates,focal-security,now 5.4.0-182.202 amd64 [installed,automatic]<br>
447,449c447,449<br>
< linux-tools-azure/now 5.15.0.1033.40~20.04.23 amd64 [installed,upgradable to: 5.15.0.1059.67~20.04.49]<br>
< linux-tools-common/focal-updates,focal-security,now 5.4.0-139.156 all [installed,upgradable to: 5.4.0-174.193]<br>
< locales/now 2.31-0ubuntu9.9 all [installed,upgradable to: 2.31-0ubuntu9.14]<br>
---<br>
> linux-tools-azure/now 5.15.0.1033.40~20.04.23 amd64 [installed,upgradable to: 5.15.0.1064.73~20.04.1]<br>
> linux-tools-common/focal-updates,focal-security,now 5.4.0-139.156 all [installed,upgradable to: 5.4.0-182.202]<br>
> locales/now 2.31-0ubuntu9.9 all [installed,upgradable to: 2.31-0ubuntu9.15]<br>
473c473<br>
< mount/focal-security,now 2.34-0.1ubuntu9.3 amd64 [installed,upgradable to: 2.34-0.1ubuntu9.4]<br>
---<br>
> mount/focal-updates,focal-security,now 2.34-0.1ubuntu9.6 amd64 [installed,automatic]<br>
499c499<br>
< passwd/now 1:4.8.1-1ubuntu5.20.04.4 amd64 [installed,upgradable to: 1:4.8.1-1ubuntu5.20.04.5]<br>
---<br>
> passwd/focal-updates,focal-security,now 1:4.8.1-1ubuntu5.20.04.5 amd64 [installed,automatic]<br>
555c555<br>
< python3-jinja2/focal,now 2.10.1-2 all [installed,upgradable to: 2.10.1-2ubuntu0.2]<br>
---<br>
> python3-jinja2/focal-updates,focal-security,now 2.10.1-2ubuntu0.2 all [installed]<br>
592,594c592,594<br>
< python3-twisted-bin/now 18.9.0-11ubuntu0.20.04.2 amd64 [installed,upgradable to: 18.9.0-11ubuntu0.20.04.3]<br>
< python3-twisted/now 18.9.0-11ubuntu0.20.04.2 all [installed,upgradable to: 18.9.0-11ubuntu0.20.04.3]<br>
< python3-update-manager/now 1:20.04.10.11 all [installed,upgradable to: 1:20.04.10.20]<br>
---<br>
> python3-twisted-bin/focal-updates,focal-security,now 18.9.0-11ubuntu0.20.04.3 amd64 [installed,automatic]<br>
> python3-twisted/focal-updates,focal-security,now 18.9.0-11ubuntu0.20.04.3 all [installed,automatic]<br>
> python3-update-manager/now 1:20.04.10.11 all [installed,upgradable to: 1:20.04.10.21]<br>
618c618<br>
< snapd/now 2.58+20.04 amd64 [installed,upgradable to: 2.58+20.04.1]<br>
---<br>
> snapd/now 2.58+20.04 amd64 [installed,upgradable to: 2.62+20.04]<br>
625,627c625,627<br>
< sudo/now 1.8.31-1ubuntu1.4 amd64 [installed,upgradable to: 1.8.31-1ubuntu1.5]<br>
< systemd-sysv/now 245.4-4ubuntu3.19 amd64 [installed,upgradable to: 245.4-4ubuntu3.23]<br>
< systemd/now 245.4-4ubuntu3.19 amd64 [installed,upgradable to: 245.4-4ubuntu3.23]<br>
---<br>
> sudo/focal-updates,focal-security,now 1.8.31-1ubuntu1.5 amd64 [installed,automatic]<br>
> systemd-sysv/focal-security,now 245.4-4ubuntu3.20 amd64 [installed,upgradable to: 245.4-4ubuntu3.23]<br>
> systemd/focal-security,now 245.4-4ubuntu3.20 amd64 [installed,upgradable to: 245.4-4ubuntu3.23]<br>
637c637<br>
< ubuntu-advantage-tools/now 27.13.5~20.04.1 amd64 [installed,upgradable to: 31.2~20.04]<br>
---<br>
> ubuntu-advantage-tools/now 27.13.5~20.04.1 amd64 [installed,upgradable to: 31.2.3~20.04]<br>
648,649c648,649<br>
< update-manager-core/now 1:20.04.10.11 all [installed,upgradable to: 1:20.04.10.20]<br>
< update-notifier-common/now 3.192.30.16 all [installed,upgradable to: 3.192.30.17]<br>
---<br>
> update-manager-core/now 1:20.04.10.11 all [installed,upgradable to: 1:20.04.10.21]<br>
> update-notifier-common/now 3.192.30.16 all [installed,upgradable to: 3.192.30.19]<br>
654,655c654,655<br>
< util-linux/focal-security,now 2.34-0.1ubuntu9.3 amd64 [installed,upgradable to: 2.34-0.1ubuntu9.4]<br>
< uuid-runtime/focal-security,now 2.34-0.1ubuntu9.3 amd64 [installed,upgradable to: 2.34-0.1ubuntu9.4]<br>
---<br>
> util-linux/now 2.34-0.1ubuntu9.3 amd64 [installed,upgradable to: 2.34-0.1ubuntu9.6]<br>
> uuid-runtime/now 2.34-0.1ubuntu9.3 amd64 [installed,upgradable to: 2.34-0.1ubuntu9.6]<br>
_______________________________________________<br>
varnish-misc mailing list<br>
<a href="mailto:varnish-misc@varnish-cache.org" rel="noreferrer" target="_blank">varnish-misc@varnish-cache.org</a><br>
<a href="https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc" rel="noreferrer noreferrer" target="_blank">https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc</a><br>
</blockquote></div>