From uday.polu at indiamart.com Thu May 2 00:21:58 2024 From: uday.polu at indiamart.com (Uday Kumar) Date: Thu, 2 May 2024 05:51:58 +0530 Subject: Append uniqueid to a http request at varnish In-Reply-To: References: Message-ID: Hello, Am I missing anything here? On Tue, Apr 30, 2024, 13:37 Uday Kumar wrote: > hello Guillaume, > > I am trying to install vmod_uuid on my centOS 7 machine > > Resource i used: > https://github.com/otto-de/libvmod-uuid/blob/5.x/INSTALL.rst > > varnish version: 5.2.1 > > I am getting below errors while running *make *command > > make[1]: Entering directory `/usr/local/src/libvmod-uuid' > Making all in src > make[2]: Entering directory `/usr/local/src/libvmod-uuid/src' > CC vmod_uuid.lo > In file included from vmod_uuid.c:35:0: > vcc_if.h:11:1: error: unknown type name ?VCL_STRING? > VCL_STRING vmod_uuid(VRT_CTX, struct vmod_priv *); > ^ > vcc_if.h:11:31: error: expected ?)? before ?struct? > VCL_STRING vmod_uuid(VRT_CTX, struct vmod_priv *); > ^ > vcc_if.h:12:1: error: unknown type name ?VCL_STRING? > VCL_STRING vmod_uuid_v1(VRT_CTX, struct vmod_priv *); > ^ > vcc_if.h:12:34: error: expected ?)? before ?struct? > VCL_STRING vmod_uuid_v1(VRT_CTX, struct vmod_priv *); > ^ > vcc_if.h:13:1: error: unknown type name ?VCL_STRING? > VCL_STRING vmod_uuid_v3(VRT_CTX, struct vmod_priv *, VCL_STRING, > ^ > vcc_if.h:13:34: error: expected ?)? before ?struct? > VCL_STRING vmod_uuid_v3(VRT_CTX, struct vmod_priv *, VCL_STRING, > ^ > vcc_if.h:15:1: error: unknown type name ?VCL_STRING? > VCL_STRING vmod_uuid_v4(VRT_CTX, struct vmod_priv *); > ^ > vcc_if.h:15:34: error: expected ?)? before ?struct? > VCL_STRING vmod_uuid_v4(VRT_CTX, struct vmod_priv *); > ^ > vcc_if.h:16:1: error: unknown type name ?VCL_STRING? > VCL_STRING vmod_uuid_v5(VRT_CTX, struct vmod_priv *, VCL_STRING, > ^ > vcc_if.h:16:34: error: expected ?)? before ?struct? > VCL_STRING vmod_uuid_v5(VRT_CTX, struct vmod_priv *, VCL_STRING, > ^ > vmod_uuid.c:48:17: error: expected ?)? before ?int? > mkuuid(VRT_CTX, int utype, uuid_t *uuid, const char *str, va_list ap) > ^ > vmod_uuid.c:76:1: error: unknown type name ?VCL_STRING? > _uuid(VRT_CTX, uuid_t *uuid, int utype, ...) > ^ > vmod_uuid.c:76:16: error: expected ?)? before ?uuid_t? > _uuid(VRT_CTX, uuid_t *uuid, int utype, ...) > ^ > vmod_uuid.c:104:21: error: expected ?)? before ?void? > free_uuids(VRT_CTX, void *priv) > ^ > vmod_uuid.c:116:39: error: array type has incomplete element type > static const struct vmod_priv_methods uuid_priv_task_methods[1] = {{ > ^ > vmod_uuid.c:117:3: error: field name not in record or union initializer > .magic = VMOD_PRIV_METHODS_MAGIC, > ^ > vmod_uuid.c:117:3: error: (near initialization for > ?uuid_priv_task_methods?) > vmod_uuid.c:117:12: error: ?VMOD_PRIV_METHODS_MAGIC? undeclared here (not > in a function) > .magic = VMOD_PRIV_METHODS_MAGIC, > ^ > vmod_uuid.c:118:3: error: field name not in record or union initializer > .type = "vmod_uuid_priv_task", > ^ > vmod_uuid.c:118:3: error: (near initialization for > ?uuid_priv_task_methods?) > vmod_uuid.c:119:3: error: field name not in record or union initializer > .fini = free_uuids > ^ > vmod_uuid.c:119:3: error: (near initialization for > ?uuid_priv_task_methods?) > vmod_uuid.c:119:11: error: ?free_uuids? undeclared here (not in a function) > .fini = free_uuids > ^ > vmod_uuid.c:123:20: error: expected ?)? before ?struct? > get_uuids(VRT_CTX, struct vmod_priv *priv, uuid_t **uuid_ns) > ^ > vmod_uuid.c:163:1: error: unknown type name ?VCL_STRING? > VCL_STRING > ^ > vmod_uuid.c:164:23: error: expected ?)? before ?struct? > vmod_uuid_v1(VRT_CTX, struct vmod_priv *priv) > ^ > vmod_uuid.c:172:1: error: unknown type name ?VCL_STRING? > VCL_STRING > ^ > vmod_uuid.c:173:23: error: expected ?)? before ?struct? > vmod_uuid_v3(VRT_CTX, struct vmod_priv *priv, VCL_STRING ns, VCL_STRING > name) > ^ > vmod_uuid.c:182:1: error: unknown type name ?VCL_STRING? > VCL_STRING > ^ > vmod_uuid.c:183:23: error: expected ?)? before ?struct? > vmod_uuid_v4(VRT_CTX, struct vmod_priv *priv) > ^ > vmod_uuid.c:191:1: error: unknown type name ?VCL_STRING? > VCL_STRING > ^ > vmod_uuid.c:192:23: error: expected ?)? before ?struct? > vmod_uuid_v5(VRT_CTX, struct vmod_priv *priv, VCL_STRING ns, VCL_STRING > name) > ^ > vmod_uuid.c:201:1: error: unknown type name ?VCL_STRING? > VCL_STRING > ^ > vmod_uuid.c:202:20: error: expected ?)? before ?struct? > vmod_uuid(VRT_CTX, struct vmod_priv *priv) > ^ > vmod_uuid.c:116:39: error: ?uuid_priv_task_methods? defined but not used > [-Werror=unused-variable] > static const struct vmod_priv_methods uuid_priv_task_methods[1] = {{ > ^ > cc1: all warnings being treated as errors > make[2]: *** [vmod_uuid.lo] Error 1 > make[2]: Leaving directory `/usr/local/src/libvmod-uuid/src' > make[1]: *** [all-recursive] Error 1 > make[1]: Leaving directory `/usr/local/src/libvmod-uuid' > make: *** [all] Error 2 > > Please help > > *Thanks & Regards,* > *Uday Kumar* > > > On Thu, Apr 25, 2024 at 6:18?AM Uday Kumar > wrote: > >> Hi Guillaume, >> >> Thanks for this reminder, I will check this and get back to you! >> >> >> *Thanks & Regards,* >> *Uday Kumar* >> >> >> On Thu, Apr 25, 2024 at 1:12?AM Guillaume Quintard < >> guillaume.quintard at gmail.com> wrote: >> >>> Hi Uday, >>> >>> I feel like we've explored this last year: >>> https://varnish-cache.org/lists/pipermail/varnish-misc/2023-May/027238.html >>> >>> I don't think the answer has changed much: vmod-uuid is your best bet >>> here. >>> >>> Please let me know if I'm missing some requirements. >>> >>> Kind regards, >>> >>> -- >>> Guillaume Quintard >>> >>> >>> On Wed, Apr 24, 2024 at 4:26?AM Uday Kumar >>> wrote: >>> >>>> Hello all, >>>> >>>> We follow below architecture in our production environment: >>>> User request ---> Varnish ----> Tomcat Backend >>>> >>>> We have a requirement of generating an unique id at varnish that can be >>>> appended to a request url. >>>> So that it can be propagated to the backend and also will be useful in >>>> tracking errors efficiently >>>> >>>> varnish version used: varnish-5.2.1 >>>> >>>> Example: >>>> original request: >>>> /search/test?q=bags&source=mobile >>>> >>>> After appending unique id [This needs to be sent to backend and to be >>>> stored in varnish logs]: >>>> /search/test?q=bags&source=mobile&uniqueid=abc123 >>>> >>>> Please help us know if there is any way to do this at varnish >>>> >>>> *Thanks & Regards,* >>>> *Uday Kumar* >>>> _______________________________________________ >>>> varnish-misc mailing list >>>> varnish-misc at varnish-cache.org >>>> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc >>>> >>> -------------- next part -------------- An HTML attachment was scrubbed... URL: From guillaume.quintard at gmail.com Thu May 2 01:19:04 2024 From: guillaume.quintard at gmail.com (Guillaume Quintard) Date: Wed, 1 May 2024 18:19:04 -0700 Subject: Append uniqueid to a http request at varnish In-Reply-To: References: Message-ID: Hi Uday, I'm not sure what went wrong, but this Dockerfile works: from centos:7 RUN \ set -ex; \ ulimit -n 10240; \ yum install -y make automake pkg-config libtool python-docutils autoconf-archive uuid-devel epel-release python-docutils; \ curl -s https://packagecloud.io/install/repositories/varnishcache/varnish5/script.rpm.sh | bash; \ yum install -y varnish-devel; \ curl -O https://raw.githubusercontent.com/varnish/toolbox/master/install-vmod/install-vmod; \ chmod +x install-vmod ;\ ./install-vmod https://github.com/otto-de/libvmod-uuid/archive/refs/heads/5.x.tar.gz Also, I know you have heard it before, but Varnish 5 is ancient, please don't use it and upgrade to something more recent, either 7.5 or to the 6.0 LTS. Let us know how it goes. -- Guillaume Quintard On Tue, Apr 30, 2024 at 1:07?AM Uday Kumar wrote: > hello Guillaume, > > I am trying to install vmod_uuid on my centOS 7 machine > > Resource i used: > https://github.com/otto-de/libvmod-uuid/blob/5.x/INSTALL.rst > > varnish version: 5.2.1 > > I am getting below errors while running *make *command > > make[1]: Entering directory `/usr/local/src/libvmod-uuid' > Making all in src > make[2]: Entering directory `/usr/local/src/libvmod-uuid/src' > CC vmod_uuid.lo > In file included from vmod_uuid.c:35:0: > vcc_if.h:11:1: error: unknown type name ?VCL_STRING? > VCL_STRING vmod_uuid(VRT_CTX, struct vmod_priv *); > ^ > vcc_if.h:11:31: error: expected ?)? before ?struct? > VCL_STRING vmod_uuid(VRT_CTX, struct vmod_priv *); > ^ > vcc_if.h:12:1: error: unknown type name ?VCL_STRING? > VCL_STRING vmod_uuid_v1(VRT_CTX, struct vmod_priv *); > ^ > vcc_if.h:12:34: error: expected ?)? before ?struct? > VCL_STRING vmod_uuid_v1(VRT_CTX, struct vmod_priv *); > ^ > vcc_if.h:13:1: error: unknown type name ?VCL_STRING? > VCL_STRING vmod_uuid_v3(VRT_CTX, struct vmod_priv *, VCL_STRING, > ^ > vcc_if.h:13:34: error: expected ?)? before ?struct? > VCL_STRING vmod_uuid_v3(VRT_CTX, struct vmod_priv *, VCL_STRING, > ^ > vcc_if.h:15:1: error: unknown type name ?VCL_STRING? > VCL_STRING vmod_uuid_v4(VRT_CTX, struct vmod_priv *); > ^ > vcc_if.h:15:34: error: expected ?)? before ?struct? > VCL_STRING vmod_uuid_v4(VRT_CTX, struct vmod_priv *); > ^ > vcc_if.h:16:1: error: unknown type name ?VCL_STRING? > VCL_STRING vmod_uuid_v5(VRT_CTX, struct vmod_priv *, VCL_STRING, > ^ > vcc_if.h:16:34: error: expected ?)? before ?struct? > VCL_STRING vmod_uuid_v5(VRT_CTX, struct vmod_priv *, VCL_STRING, > ^ > vmod_uuid.c:48:17: error: expected ?)? before ?int? > mkuuid(VRT_CTX, int utype, uuid_t *uuid, const char *str, va_list ap) > ^ > vmod_uuid.c:76:1: error: unknown type name ?VCL_STRING? > _uuid(VRT_CTX, uuid_t *uuid, int utype, ...) > ^ > vmod_uuid.c:76:16: error: expected ?)? before ?uuid_t? > _uuid(VRT_CTX, uuid_t *uuid, int utype, ...) > ^ > vmod_uuid.c:104:21: error: expected ?)? before ?void? > free_uuids(VRT_CTX, void *priv) > ^ > vmod_uuid.c:116:39: error: array type has incomplete element type > static const struct vmod_priv_methods uuid_priv_task_methods[1] = {{ > ^ > vmod_uuid.c:117:3: error: field name not in record or union initializer > .magic = VMOD_PRIV_METHODS_MAGIC, > ^ > vmod_uuid.c:117:3: error: (near initialization for > ?uuid_priv_task_methods?) > vmod_uuid.c:117:12: error: ?VMOD_PRIV_METHODS_MAGIC? undeclared here (not > in a function) > .magic = VMOD_PRIV_METHODS_MAGIC, > ^ > vmod_uuid.c:118:3: error: field name not in record or union initializer > .type = "vmod_uuid_priv_task", > ^ > vmod_uuid.c:118:3: error: (near initialization for > ?uuid_priv_task_methods?) > vmod_uuid.c:119:3: error: field name not in record or union initializer > .fini = free_uuids > ^ > vmod_uuid.c:119:3: error: (near initialization for > ?uuid_priv_task_methods?) > vmod_uuid.c:119:11: error: ?free_uuids? undeclared here (not in a function) > .fini = free_uuids > ^ > vmod_uuid.c:123:20: error: expected ?)? before ?struct? > get_uuids(VRT_CTX, struct vmod_priv *priv, uuid_t **uuid_ns) > ^ > vmod_uuid.c:163:1: error: unknown type name ?VCL_STRING? > VCL_STRING > ^ > vmod_uuid.c:164:23: error: expected ?)? before ?struct? > vmod_uuid_v1(VRT_CTX, struct vmod_priv *priv) > ^ > vmod_uuid.c:172:1: error: unknown type name ?VCL_STRING? > VCL_STRING > ^ > vmod_uuid.c:173:23: error: expected ?)? before ?struct? > vmod_uuid_v3(VRT_CTX, struct vmod_priv *priv, VCL_STRING ns, VCL_STRING > name) > ^ > vmod_uuid.c:182:1: error: unknown type name ?VCL_STRING? > VCL_STRING > ^ > vmod_uuid.c:183:23: error: expected ?)? before ?struct? > vmod_uuid_v4(VRT_CTX, struct vmod_priv *priv) > ^ > vmod_uuid.c:191:1: error: unknown type name ?VCL_STRING? > VCL_STRING > ^ > vmod_uuid.c:192:23: error: expected ?)? before ?struct? > vmod_uuid_v5(VRT_CTX, struct vmod_priv *priv, VCL_STRING ns, VCL_STRING > name) > ^ > vmod_uuid.c:201:1: error: unknown type name ?VCL_STRING? > VCL_STRING > ^ > vmod_uuid.c:202:20: error: expected ?)? before ?struct? > vmod_uuid(VRT_CTX, struct vmod_priv *priv) > ^ > vmod_uuid.c:116:39: error: ?uuid_priv_task_methods? defined but not used > [-Werror=unused-variable] > static const struct vmod_priv_methods uuid_priv_task_methods[1] = {{ > ^ > cc1: all warnings being treated as errors > make[2]: *** [vmod_uuid.lo] Error 1 > make[2]: Leaving directory `/usr/local/src/libvmod-uuid/src' > make[1]: *** [all-recursive] Error 1 > make[1]: Leaving directory `/usr/local/src/libvmod-uuid' > make: *** [all] Error 2 > > Please help > > *Thanks & Regards,* > *Uday Kumar* > > > On Thu, Apr 25, 2024 at 6:18?AM Uday Kumar > wrote: > >> Hi Guillaume, >> >> Thanks for this reminder, I will check this and get back to you! >> >> >> *Thanks & Regards,* >> *Uday Kumar* >> >> >> On Thu, Apr 25, 2024 at 1:12?AM Guillaume Quintard < >> guillaume.quintard at gmail.com> wrote: >> >>> Hi Uday, >>> >>> I feel like we've explored this last year: >>> https://varnish-cache.org/lists/pipermail/varnish-misc/2023-May/027238.html >>> >>> I don't think the answer has changed much: vmod-uuid is your best bet >>> here. >>> >>> Please let me know if I'm missing some requirements. >>> >>> Kind regards, >>> >>> -- >>> Guillaume Quintard >>> >>> >>> On Wed, Apr 24, 2024 at 4:26?AM Uday Kumar >>> wrote: >>> >>>> Hello all, >>>> >>>> We follow below architecture in our production environment: >>>> User request ---> Varnish ----> Tomcat Backend >>>> >>>> We have a requirement of generating an unique id at varnish that can be >>>> appended to a request url. >>>> So that it can be propagated to the backend and also will be useful in >>>> tracking errors efficiently >>>> >>>> varnish version used: varnish-5.2.1 >>>> >>>> Example: >>>> original request: >>>> /search/test?q=bags&source=mobile >>>> >>>> After appending unique id [This needs to be sent to backend and to be >>>> stored in varnish logs]: >>>> /search/test?q=bags&source=mobile&uniqueid=abc123 >>>> >>>> Please help us know if there is any way to do this at varnish >>>> >>>> *Thanks & Regards,* >>>> *Uday Kumar* >>>> _______________________________________________ >>>> varnish-misc mailing list >>>> varnish-misc at varnish-cache.org >>>> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc >>>> >>> -------------- next part -------------- An HTML attachment was scrubbed... URL: From batanun at hotmail.com Thu May 16 09:17:31 2024 From: batanun at hotmail.com (Batanun B) Date: Thu, 16 May 2024 09:17:31 +0000 Subject: Varnish suddenly started using much more memory Message-ID: Hi, About two weeks ago we deployed some minor changes to our Varnish servers in production, and after that we have noticed a big change in the memory that Varnish consumes. Before the deploy, the amount of available memory on the servers were very stable, around 25 GB, for months on end. After the deploy, the amount of available memory dropped below 25 GB within 6 hours, and is dropping about 1 GB more each day, with no indication that it will level out before hitting rock bottom. There was no change in traffic patterns during the time of the deploy. And we didn't change any OS or Varnish configuration. The deplow consisted only of trivial VCL changes, like changing the backend probe url to a dedicated healthcheck endpoint, and tweaking the ttl for a minor resource. Nothing of which could explain this massive change in memory usage. We have configured varnish with "-s default=malloc,12G -s large=malloc,8G", where the combined 20GB is about 60% of the total server RAM of 32GB. This is below the recommended 75% maximum I've seen in many places. Currently Varnish uses about 73% of the server memory, or 23GB (the RES column in htop). The default storage uses about 10 GB (SMA.default.g_bytes), while the large storage uses 8 GB. And the transient storage is currently about 2 MB (SMA.Transient.g_bytes). In total this results in about 18 GB. So what is that additional 5 GB used for? How can I troubleshoot that? And, more importantly, what could possibly explain this sudden change? The Ubuntu version stayed the same (20.04.5 LTS), and the Varnish version too (6.0.11-1~focal), as well as varnish-modules (0.15.1). I notice some differences in some installed packages of the servers, but nothing that stands out to me (but I'm no linux expert). Regards -------------- next part -------------- An HTML attachment was scrubbed... URL: From guillaume.quintard at gmail.com Thu May 16 23:01:17 2024 From: guillaume.quintard at gmail.com (Guillaume Quintard) Date: Thu, 16 May 2024 16:01:17 -0700 Subject: Varnish suddenly started using much more memory In-Reply-To: References: Message-ID: Hi, I feel like the answer is there, somewhere. You said that the deploy changed something, but that it can't possibly be the deploy. I'm going to bet that it's the deploy. Most likely you changed something that messed up the willingness to cache, or your TTL. First, check the difference in passes, if they are about the same, look for hit-for-misses, and lastly, look at how long Varnish is trying to cache the average object. I'm pretty one of those changed. That being said, the memory shouldn't explode like that, which packages are you using? -- Guillaume Quintard On Thu, May 16, 2024 at 2:19?AM Batanun B wrote: > Hi, > > About two weeks ago we deployed some minor changes to our Varnish servers > in production, and after that we have noticed a big change in the memory > that Varnish consumes. > > Before the deploy, the amount of available memory on the servers were very > stable, around 25 GB, for months on end. After the deploy, the amount of > available memory dropped below 25 GB within 6 hours, and is dropping about > 1 GB more each day, with no indication that it will level out before > hitting rock bottom. > > There was no change in traffic patterns during the time of the deploy. And > we didn't change any OS or Varnish configuration. The deplow consisted only > of trivial VCL changes, like changing the backend probe url to a dedicated > healthcheck endpoint, and tweaking the ttl for a minor resource. Nothing of > which could explain this massive change in memory usage. > > We have configured varnish with "-s default=malloc,12G -s > large=malloc,8G", where the combined 20GB is about 60% of the total server > RAM of 32GB. This is below the recommended 75% maximum I've seen in many > places. > > Currently Varnish uses about 73% of the server memory, or 23GB (the RES > column in htop). The default storage uses about 10 GB > (SMA.default.g_bytes), while the large storage uses 8 GB. And the transient > storage is currently about 2 MB (SMA.Transient.g_bytes). In total this > results in about 18 GB. So what is that additional 5 GB used for? How can I > troubleshoot that? > > And, more importantly, what could possibly explain this sudden change? > > The Ubuntu version stayed the same (20.04.5 LTS), and the Varnish version > too (6.0.11-1~focal), as well as varnish-modules (0.15.1). I notice some > differences in some installed packages of the servers, but nothing that > stands out to me (but I'm no linux expert). > > Regards > _______________________________________________ > varnish-misc mailing list > varnish-misc at varnish-cache.org > https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc > -------------- next part -------------- An HTML attachment was scrubbed... URL: From batanun at hotmail.com Fri May 17 13:15:10 2024 From: batanun at hotmail.com (Batanun B) Date: Fri, 17 May 2024 13:15:10 +0000 Subject: Varnish suddenly started using much more memory In-Reply-To: References: Message-ID: Hi, Naturally, I can't be certain that the "in my mind" trivial VCL changes can't be the culprit. But I just can't see the logic in those changes causing this massive change in memory usage. But I'll summarize the changes here, and maybe you can identify a suspect: * Modified the xkey header used by the xkey vmod, adding the id of the current website * Modified the TTL, from 1w to 1h, for a specific type of resource existing in maybe 20 versions (ie different urls), each being about 5 kB in size * Modified the backend probe url, from the startpage (ie full html) to a dedicated healthcheck endpoint (much smaller footprint, and much quicker) That's it. That's all the VCL changes we made in that deployment. And, like I said, we did no changes in the OS or Varnish config. > check the difference in passes, if they are about the same, look for hit-for-misses, We don't have those statistics from the old server, so I can't do a comparison. But here are the current statistics: MAIN.s_pass ? ? ? ? ? ? ? ? ? ? ? 180721 ? ? ? ? 0.14 Total pass-ed requests seen MAIN.cache_hitpass ? ? ? ? ? ? ? ? ? ? 0 ? ? ? ? 0.00 Cache hits for pass. MAIN.cache_hit ? ? ? ? ? ? ? ? ? 3718468 ? ? ? ? 2.86 Cache hits MAIN.cache_hit_grace ? ? ? ? ? ? ? 53903 ? ? ? ? 0.04 Cache grace hits MAIN.cache_hitpass ? ? ? ? ? ? ? ? ? ? 0 ? ? ? ? 0.00 Cache hits for pass. MAIN.cache_hitmiss ? ? ? ? ? ? ? ? ?1129 ? ? ? ? 0.00 Cache hits for miss. > and lastly, look at how long Varnish is trying to cache the average object. I'm not sure how I do that. Is there a varnishstat counter I can look at? > which packages are you using? Instead of giving you the full list, I guess it makes more sense to just list the one that differ. Below is a diff output of "apt list --installed" of before and after the deploy. Regards 2c2 < accountsservice/now 0.6.55-0ubuntu12~20.04.5 amd64 [installed,upgradable to: 0.6.55-0ubuntu12~20.04.7] --- > accountsservice/focal-updates,focal-security,now 0.6.55-0ubuntu12~20.04.7 amd64 [installed,automatic] 23,25c23,25 < bind9-dnsutils/now 1:9.16.1-0ubuntu2.12 amd64 [installed,upgradable to: 1:9.16.48-0ubuntu0.20.04.1] < bind9-host/now 1:9.16.1-0ubuntu2.12 amd64 [installed,upgradable to: 1:9.16.48-0ubuntu0.20.04.1] < bind9-libs/now 1:9.16.1-0ubuntu2.12 amd64 [installed,upgradable to: 1:9.16.48-0ubuntu0.20.04.1] --- > bind9-dnsutils/focal-updates,focal-security,now 1:9.16.48-0ubuntu0.20.04.1 amd64 [installed,automatic] > bind9-host/focal-updates,focal-security,now 1:9.16.48-0ubuntu0.20.04.1 amd64 [installed,automatic] > bind9-libs/focal-updates,focal-security,now 1:9.16.48-0ubuntu0.20.04.1 amd64 [installed,automatic] 31c31 < bsdutils/focal-security,now 1:2.34-0.1ubuntu9.3 amd64 [installed,upgradable to: 1:2.34-0.1ubuntu9.4] --- > bsdutils/now 1:2.34-0.1ubuntu9.3 amd64 [installed,upgradable to: 1:2.34-0.1ubuntu9.6] 41c41 < cloud-init/now 22.4.2-0ubuntu0~20.04.2 all [installed,upgradable to: 23.4.4-0ubuntu0~20.04.1] --- > cloud-init/now 22.4.2-0ubuntu0~20.04.2 all [installed,upgradable to: 24.1.3-0ubuntu1~20.04.1] 48c48 < cpio/focal-updates,focal-security,now 2.13+dfsg-2ubuntu0.3 amd64 [installed,automatic] --- > cpio/now 2.13+dfsg-2ubuntu0.3 amd64 [installed,upgradable to: 2.13+dfsg-2ubuntu0.4] 56c56 < curl/focal-updates,focal-security,now 7.68.0-1ubuntu2.21 amd64 [installed] --- > curl/focal-updates,focal-security,now 7.68.0-1ubuntu2.22 amd64 [installed] 68c68 < distro-info-data/now 0.43ubuntu1.11 all [installed,upgradable to: 0.43ubuntu1.15] --- > distro-info-data/now 0.43ubuntu1.11 all [installed,upgradable to: 0.43ubuntu1.16] 82c82 < fdisk/focal-security,now 2.34-0.1ubuntu9.3 amd64 [installed,upgradable to: 2.34-0.1ubuntu9.4] --- > fdisk/now 2.34-0.1ubuntu9.3 amd64 [installed,upgradable to: 2.34-0.1ubuntu9.6] 119,120c119,120 < grub-efi-amd64-bin/now 2.06-2ubuntu14.1 amd64 [installed,upgradable to: 2.06-2ubuntu14.4] < grub-efi-amd64-signed/now 1.187.3~20.04.1+2.06-2ubuntu14.1 amd64 [installed,upgradable to: 1.187.6~20.04.1+2.06-2ubuntu14.4] --- > grub-efi-amd64-bin/focal-updates,focal-security,now 2.06-2ubuntu14.4 amd64 [installed] > grub-efi-amd64-signed/focal-updates,focal-security,now 1.187.6~20.04.1+2.06-2ubuntu14.4 amd64 [installed] 148c148 < klibc-utils/focal-updates,focal-security,now 2.0.7-1ubuntu5.1 amd64 [installed,automatic] --- > klibc-utils/focal-updates,focal-security,now 2.0.7-1ubuntu5.2 amd64 [installed,automatic] 152c152 < landscape-common/focal-updates,now 19.12-0ubuntu4.3 amd64 [installed,automatic] --- > landscape-common/now 19.12-0ubuntu4.3 amd64 [installed,upgradable to: 23.02-0ubuntu1~20.04.2] 154,155c154,155 < less/now 551-1ubuntu0.1 amd64 [installed,upgradable to: 551-1ubuntu0.2] < libaccountsservice0/now 0.6.55-0ubuntu12~20.04.5 amd64 [installed,upgradable to: 0.6.55-0ubuntu12~20.04.7] --- > less/now 551-1ubuntu0.1 amd64 [installed,upgradable to: 551-1ubuntu0.3] > libaccountsservice0/focal-updates,focal-security,now 0.6.55-0ubuntu12~20.04.7 amd64 [installed,automatic] 175c175 < libblkid1/focal-security,now 2.34-0.1ubuntu9.3 amd64 [installed,upgradable to: 2.34-0.1ubuntu9.4] --- > libblkid1/now 2.34-0.1ubuntu9.3 amd64 [installed,upgradable to: 2.34-0.1ubuntu9.6] 187,190c187,190 < libc-bin/now 2.31-0ubuntu9.9 amd64 [installed,upgradable to: 2.31-0ubuntu9.14] < libc-dev-bin/focal-updates,focal-security,now 2.31-0ubuntu9.14 amd64 [installed,automatic] < libc6-dev/focal-updates,focal-security,now 2.31-0ubuntu9.14 amd64 [installed,automatic] < libc6/focal-updates,focal-security,now 2.31-0ubuntu9.14 amd64 [installed,automatic] --- > libc-bin/now 2.31-0ubuntu9.9 amd64 [installed,upgradable to: 2.31-0ubuntu9.15] > libc-dev-bin/focal-updates,focal-security,now 2.31-0ubuntu9.15 amd64 [installed,automatic] > libc6-dev/focal-updates,focal-security,now 2.31-0ubuntu9.15 amd64 [installed,automatic] > libc6/focal-updates,focal-security,now 2.31-0ubuntu9.15 amd64 [installed,automatic] 203,204c203,204 < libcurl3-gnutls/now 7.68.0-1ubuntu2.15 amd64 [installed,upgradable to: 7.68.0-1ubuntu2.21] < libcurl4/focal-updates,focal-security,now 7.68.0-1ubuntu2.21 amd64 [installed,automatic] --- > libcurl3-gnutls/now 7.68.0-1ubuntu2.15 amd64 [installed,upgradable to: 7.68.0-1ubuntu2.22] > libcurl4/focal-updates,focal-security,now 7.68.0-1ubuntu2.22 amd64 [installed,automatic] 227c227 < libfdisk1/focal-security,now 2.34-0.1ubuntu9.3 amd64 [installed,upgradable to: 2.34-0.1ubuntu9.4] --- > libfdisk1/now 2.34-0.1ubuntu9.3 amd64 [installed,upgradable to: 2.34-0.1ubuntu9.6] 244,246c244,246 < libglib2.0-0/now 2.64.6-1~ubuntu20.04.4 amd64 [installed,upgradable to: 2.64.6-1~ubuntu20.04.6] < libglib2.0-bin/now 2.64.6-1~ubuntu20.04.4 amd64 [installed,upgradable to: 2.64.6-1~ubuntu20.04.6] < libglib2.0-data/now 2.64.6-1~ubuntu20.04.4 all [installed,upgradable to: 2.64.6-1~ubuntu20.04.6] --- > libglib2.0-0/now 2.64.6-1~ubuntu20.04.4 amd64 [installed,upgradable to: 2.64.6-1~ubuntu20.04.7] > libglib2.0-bin/now 2.64.6-1~ubuntu20.04.4 amd64 [installed,upgradable to: 2.64.6-1~ubuntu20.04.7] > libglib2.0-data/now 2.64.6-1~ubuntu20.04.4 all [installed,upgradable to: 2.64.6-1~ubuntu20.04.7] 248c248 < libgnutls30/now 3.6.13-2ubuntu1.7 amd64 [installed,upgradable to: 3.6.13-2ubuntu1.10] --- > libgnutls30/now 3.6.13-2ubuntu1.7 amd64 [installed,upgradable to: 3.6.13-2ubuntu1.11] 278c278 < libklibc/focal-updates,focal-security,now 2.0.7-1ubuntu5.1 amd64 [installed,automatic] --- > libklibc/focal-updates,focal-security,now 2.0.7-1ubuntu5.2 amd64 [installed,automatic] 284c284 < libldap-2.4-2/now 2.4.49+dfsg-2ubuntu1.9 amd64 [installed,upgradable to: 2.4.49+dfsg-2ubuntu1.10] --- > libldap-2.4-2/focal-updates,focal-security,now 2.4.49+dfsg-2ubuntu1.10 amd64 [installed,automatic] 301c301 < libmount1/focal-security,now 2.34-0.1ubuntu9.3 amd64 [installed,upgradable to: 2.34-0.1ubuntu9.4] --- > libmount1/now 2.34-0.1ubuntu9.3 amd64 [installed,upgradable to: 2.34-0.1ubuntu9.6] 306,307c306,307 < libncurses6/focal,now 6.2-0ubuntu2 amd64 [installed,upgradable to: 6.2-0ubuntu2.1] < libncursesw6/focal,now 6.2-0ubuntu2 amd64 [installed,upgradable to: 6.2-0ubuntu2.1] --- > libncurses6/focal-updates,focal-security,now 6.2-0ubuntu2.1 amd64 [installed,automatic] > libncursesw6/focal-updates,focal-security,now 6.2-0ubuntu2.1 amd64 [installed,automatic] 314c314 < libnghttp2-14/focal,now 1.40.0-1build1 amd64 [installed,upgradable to: 1.40.0-1ubuntu0.2] --- > libnghttp2-14/focal,now 1.40.0-1build1 amd64 [installed,upgradable to: 1.40.0-1ubuntu0.3] 316,318c316,318 < libnspr4/focal,now 2:4.25-1 amd64 [installed,automatic] < libnss-systemd/now 245.4-4ubuntu3.19 amd64 [installed,upgradable to: 245.4-4ubuntu3.23] < libnss3/now 2:3.49.1-1ubuntu1.8 amd64 [installed,upgradable to: 2:3.49.1-1ubuntu1.9] --- > libnspr4/focal,now 2:4.25-1 amd64 [installed,upgradable to: 2:4.35-0ubuntu0.20.04.1] > libnss-systemd/focal-security,now 245.4-4ubuntu3.20 amd64 [installed,upgradable to: 245.4-4ubuntu3.23] > libnss3/now 2:3.49.1-1ubuntu1.8 amd64 [installed,upgradable to: 2:3.98-0ubuntu0.20.04.2] 325,328c325,328 < libpam-modules-bin/now 1.3.1-5ubuntu4.6 amd64 [installed,upgradable to: 1.3.1-5ubuntu4.7] < libpam-modules/now 1.3.1-5ubuntu4.6 amd64 [installed,upgradable to: 1.3.1-5ubuntu4.7] < libpam-runtime/now 1.3.1-5ubuntu4.6 all [installed,upgradable to: 1.3.1-5ubuntu4.7] < libpam-systemd/now 245.4-4ubuntu3.19 amd64 [installed,upgradable to: 245.4-4ubuntu3.23] --- > libpam-modules-bin/focal-updates,focal-security,now 1.3.1-5ubuntu4.7 amd64 [installed,automatic] > libpam-modules/focal-updates,focal-security,now 1.3.1-5ubuntu4.7 amd64 [installed,automatic] > libpam-runtime/focal-updates,focal-security,now 1.3.1-5ubuntu4.7 all [installed,automatic] > libpam-systemd/focal-security,now 245.4-4ubuntu3.20 amd64 [installed,upgradable to: 245.4-4ubuntu3.23] 347c347 < libprocps8/now 2:3.3.16-1ubuntu2.3 amd64 [installed,upgradable to: 2:3.3.16-1ubuntu2.4] --- > libprocps8/focal-updates,focal-security,now 2:3.3.16-1ubuntu2.4 amd64 [installed,automatic] 375c375 < libsmartcols1/focal-security,now 2.34-0.1ubuntu9.3 amd64 [installed,upgradable to: 2.34-0.1ubuntu9.4] --- > libsmartcols1/now 2.34-0.1ubuntu9.3 amd64 [installed,upgradable to: 2.34-0.1ubuntu9.6] 386c386 < libsystemd0/now 245.4-4ubuntu3.19 amd64 [installed,upgradable to: 245.4-4ubuntu3.23] --- > libsystemd0/focal-security,now 245.4-4ubuntu3.20 amd64 [installed,upgradable to: 245.4-4ubuntu3.23] 394c394 < libtinfo6/focal,now 6.2-0ubuntu2 amd64 [installed,upgradable to: 6.2-0ubuntu2.1] --- > libtinfo6/focal-updates,focal-security,now 6.2-0ubuntu2.1 amd64 [installed,automatic] 407c407 < libuuid1/focal-security,now 2.34-0.1ubuntu9.3 amd64 [installed,upgradable to: 2.34-0.1ubuntu9.4] --- > libuuid1/now 2.34-0.1ubuntu9.3 amd64 [installed,upgradable to: 2.34-0.1ubuntu9.6] 434c434 < linux-azure/now 5.15.0.1033.40~20.04.23 amd64 [installed,upgradable to: 5.15.0.1059.67~20.04.49] --- > linux-azure/now 5.15.0.1033.40~20.04.23 amd64 [installed,upgradable to: 5.15.0.1064.73~20.04.1] 438,439c438,439 < linux-cloud-tools-azure/now 5.15.0.1033.40~20.04.23 amd64 [installed,upgradable to: 5.15.0.1059.67~20.04.49] < linux-cloud-tools-common/focal-updates,focal-security,now 5.4.0-139.156 all [installed,upgradable to: 5.4.0-174.193] --- > linux-cloud-tools-azure/now 5.15.0.1033.40~20.04.23 amd64 [installed,upgradable to: 5.15.0.1064.73~20.04.1] > linux-cloud-tools-common/focal-updates,focal-security,now 5.4.0-182.202 all [installed] 441c441 < linux-headers-azure/now 5.15.0.1033.40~20.04.23 amd64 [installed,upgradable to: 5.15.0.1059.67~20.04.49] --- > linux-headers-azure/now 5.15.0.1033.40~20.04.23 amd64 [installed,upgradable to: 5.15.0.1064.73~20.04.1] 443,444c443,444 < linux-image-azure/now 5.15.0.1033.40~20.04.23 amd64 [installed,upgradable to: 5.15.0.1059.67~20.04.49] < linux-libc-dev/focal-updates,focal-security,now 5.4.0-174.193 amd64 [installed,automatic] --- > linux-image-azure/now 5.15.0.1033.40~20.04.23 amd64 [installed,upgradable to: 5.15.0.1064.73~20.04.1] > linux-libc-dev/focal-updates,focal-security,now 5.4.0-182.202 amd64 [installed,automatic] 447,449c447,449 < linux-tools-azure/now 5.15.0.1033.40~20.04.23 amd64 [installed,upgradable to: 5.15.0.1059.67~20.04.49] < linux-tools-common/focal-updates,focal-security,now 5.4.0-139.156 all [installed,upgradable to: 5.4.0-174.193] < locales/now 2.31-0ubuntu9.9 all [installed,upgradable to: 2.31-0ubuntu9.14] --- > linux-tools-azure/now 5.15.0.1033.40~20.04.23 amd64 [installed,upgradable to: 5.15.0.1064.73~20.04.1] > linux-tools-common/focal-updates,focal-security,now 5.4.0-139.156 all [installed,upgradable to: 5.4.0-182.202] > locales/now 2.31-0ubuntu9.9 all [installed,upgradable to: 2.31-0ubuntu9.15] 473c473 < mount/focal-security,now 2.34-0.1ubuntu9.3 amd64 [installed,upgradable to: 2.34-0.1ubuntu9.4] --- > mount/focal-updates,focal-security,now 2.34-0.1ubuntu9.6 amd64 [installed,automatic] 499c499 < passwd/now 1:4.8.1-1ubuntu5.20.04.4 amd64 [installed,upgradable to: 1:4.8.1-1ubuntu5.20.04.5] --- > passwd/focal-updates,focal-security,now 1:4.8.1-1ubuntu5.20.04.5 amd64 [installed,automatic] 555c555 < python3-jinja2/focal,now 2.10.1-2 all [installed,upgradable to: 2.10.1-2ubuntu0.2] --- > python3-jinja2/focal-updates,focal-security,now 2.10.1-2ubuntu0.2 all [installed] 592,594c592,594 < python3-twisted-bin/now 18.9.0-11ubuntu0.20.04.2 amd64 [installed,upgradable to: 18.9.0-11ubuntu0.20.04.3] < python3-twisted/now 18.9.0-11ubuntu0.20.04.2 all [installed,upgradable to: 18.9.0-11ubuntu0.20.04.3] < python3-update-manager/now 1:20.04.10.11 all [installed,upgradable to: 1:20.04.10.20] --- > python3-twisted-bin/focal-updates,focal-security,now 18.9.0-11ubuntu0.20.04.3 amd64 [installed,automatic] > python3-twisted/focal-updates,focal-security,now 18.9.0-11ubuntu0.20.04.3 all [installed,automatic] > python3-update-manager/now 1:20.04.10.11 all [installed,upgradable to: 1:20.04.10.21] 618c618 < snapd/now 2.58+20.04 amd64 [installed,upgradable to: 2.58+20.04.1] --- > snapd/now 2.58+20.04 amd64 [installed,upgradable to: 2.62+20.04] 625,627c625,627 < sudo/now 1.8.31-1ubuntu1.4 amd64 [installed,upgradable to: 1.8.31-1ubuntu1.5] < systemd-sysv/now 245.4-4ubuntu3.19 amd64 [installed,upgradable to: 245.4-4ubuntu3.23] < systemd/now 245.4-4ubuntu3.19 amd64 [installed,upgradable to: 245.4-4ubuntu3.23] --- > sudo/focal-updates,focal-security,now 1.8.31-1ubuntu1.5 amd64 [installed,automatic] > systemd-sysv/focal-security,now 245.4-4ubuntu3.20 amd64 [installed,upgradable to: 245.4-4ubuntu3.23] > systemd/focal-security,now 245.4-4ubuntu3.20 amd64 [installed,upgradable to: 245.4-4ubuntu3.23] 637c637 < ubuntu-advantage-tools/now 27.13.5~20.04.1 amd64 [installed,upgradable to: 31.2~20.04] --- > ubuntu-advantage-tools/now 27.13.5~20.04.1 amd64 [installed,upgradable to: 31.2.3~20.04] 648,649c648,649 < update-manager-core/now 1:20.04.10.11 all [installed,upgradable to: 1:20.04.10.20] < update-notifier-common/now 3.192.30.16 all [installed,upgradable to: 3.192.30.17] --- > update-manager-core/now 1:20.04.10.11 all [installed,upgradable to: 1:20.04.10.21] > update-notifier-common/now 3.192.30.16 all [installed,upgradable to: 3.192.30.19] 654,655c654,655 < util-linux/focal-security,now 2.34-0.1ubuntu9.3 amd64 [installed,upgradable to: 2.34-0.1ubuntu9.4] < uuid-runtime/focal-security,now 2.34-0.1ubuntu9.3 amd64 [installed,upgradable to: 2.34-0.1ubuntu9.4] --- > util-linux/now 2.34-0.1ubuntu9.3 amd64 [installed,upgradable to: 2.34-0.1ubuntu9.6] > uuid-runtime/now 2.34-0.1ubuntu9.3 amd64 [installed,upgradable to: 2.34-0.1ubuntu9.6] From guillaume.quintard at gmail.com Sat May 18 18:01:49 2024 From: guillaume.quintard at gmail.com (Guillaume Quintard) Date: Sat, 18 May 2024 11:01:49 -0700 Subject: Varnish suddenly started using much more memory In-Reply-To: References: Message-ID: Sorry, I should have been clearer, I meant: where are the varnish packages coming from? Are they from the official repositories, from https://packagecloud.io/varnishcache/ or built from source maybe? If you don't have old metrics (you should really invest some time in something like prometheus, it would probably have made the issue obvious), then we can't really compare anything. Is there any chance you can run the old version on the server to explore the differences? Two extra questions: - what's the output of: varnishstat -1 -f '*g_bytes' - have you tweaked any workspaces/thread parameters? Cheers, On Fri, May 17, 2024, 06:17 Batanun B wrote: > Hi, > > Naturally, I can't be certain that the "in my mind" trivial VCL changes > can't be the culprit. But I just can't see the logic in those changes > causing this massive change in memory usage. But I'll summarize the changes > here, and maybe you can identify a suspect: > > * Modified the xkey header used by the xkey vmod, adding the id of the > current website > * Modified the TTL, from 1w to 1h, for a specific type of resource > existing in maybe 20 versions (ie different urls), each being about 5 kB in > size > * Modified the backend probe url, from the startpage (ie full html) to a > dedicated healthcheck endpoint (much smaller footprint, and much quicker) > > That's it. That's all the VCL changes we made in that deployment. And, > like I said, we did no changes in the OS or Varnish config. > > > > check the difference in passes, if they are about the same, look for > hit-for-misses, > > We don't have those statistics from the old server, so I can't do a > comparison. But here are the current statistics: > > MAIN.s_pass 180721 0.14 Total pass-ed > requests seen > MAIN.cache_hitpass 0 0.00 Cache hits for pass. > MAIN.cache_hit 3718468 2.86 Cache hits > MAIN.cache_hit_grace 53903 0.04 Cache grace hits > MAIN.cache_hitpass 0 0.00 Cache hits for pass. > MAIN.cache_hitmiss 1129 0.00 Cache hits for miss. > > > > and lastly, look at how long Varnish is trying to cache the average > object. > > I'm not sure how I do that. Is there a varnishstat counter I can look at? > > > which packages are you using? > > Instead of giving you the full list, I guess it makes more sense to just > list the one that differ. Below is a diff output of "apt list --installed" > of before and after the deploy. > > Regards > > > 2c2 > < accountsservice/now 0.6.55-0ubuntu12~20.04.5 amd64 [installed,upgradable > to: 0.6.55-0ubuntu12~20.04.7] > --- > > accountsservice/focal-updates,focal-security,now > 0.6.55-0ubuntu12~20.04.7 amd64 [installed,automatic] > 23,25c23,25 > < bind9-dnsutils/now 1:9.16.1-0ubuntu2.12 amd64 [installed,upgradable to: > 1:9.16.48-0ubuntu0.20.04.1] > < bind9-host/now 1:9.16.1-0ubuntu2.12 amd64 [installed,upgradable to: > 1:9.16.48-0ubuntu0.20.04.1] > < bind9-libs/now 1:9.16.1-0ubuntu2.12 amd64 [installed,upgradable to: > 1:9.16.48-0ubuntu0.20.04.1] > --- > > bind9-dnsutils/focal-updates,focal-security,now > 1:9.16.48-0ubuntu0.20.04.1 amd64 [installed,automatic] > > bind9-host/focal-updates,focal-security,now 1:9.16.48-0ubuntu0.20.04.1 > amd64 [installed,automatic] > > bind9-libs/focal-updates,focal-security,now 1:9.16.48-0ubuntu0.20.04.1 > amd64 [installed,automatic] > 31c31 > < bsdutils/focal-security,now 1:2.34-0.1ubuntu9.3 amd64 > [installed,upgradable to: 1:2.34-0.1ubuntu9.4] > --- > > bsdutils/now 1:2.34-0.1ubuntu9.3 amd64 [installed,upgradable to: > 1:2.34-0.1ubuntu9.6] > 41c41 > < cloud-init/now 22.4.2-0ubuntu0~20.04.2 all [installed,upgradable to: > 23.4.4-0ubuntu0~20.04.1] > --- > > cloud-init/now 22.4.2-0ubuntu0~20.04.2 all [installed,upgradable to: > 24.1.3-0ubuntu1~20.04.1] > 48c48 > < cpio/focal-updates,focal-security,now 2.13+dfsg-2ubuntu0.3 amd64 > [installed,automatic] > --- > > cpio/now 2.13+dfsg-2ubuntu0.3 amd64 [installed,upgradable to: > 2.13+dfsg-2ubuntu0.4] > 56c56 > < curl/focal-updates,focal-security,now 7.68.0-1ubuntu2.21 amd64 > [installed] > --- > > curl/focal-updates,focal-security,now 7.68.0-1ubuntu2.22 amd64 > [installed] > 68c68 > < distro-info-data/now 0.43ubuntu1.11 all [installed,upgradable to: > 0.43ubuntu1.15] > --- > > distro-info-data/now 0.43ubuntu1.11 all [installed,upgradable to: > 0.43ubuntu1.16] > 82c82 > < fdisk/focal-security,now 2.34-0.1ubuntu9.3 amd64 [installed,upgradable > to: 2.34-0.1ubuntu9.4] > --- > > fdisk/now 2.34-0.1ubuntu9.3 amd64 [installed,upgradable to: > 2.34-0.1ubuntu9.6] > 119,120c119,120 > < grub-efi-amd64-bin/now 2.06-2ubuntu14.1 amd64 [installed,upgradable to: > 2.06-2ubuntu14.4] > < grub-efi-amd64-signed/now 1.187.3~20.04.1+2.06-2ubuntu14.1 amd64 > [installed,upgradable to: 1.187.6~20.04.1+2.06-2ubuntu14.4] > --- > > grub-efi-amd64-bin/focal-updates,focal-security,now 2.06-2ubuntu14.4 > amd64 [installed] > > grub-efi-amd64-signed/focal-updates,focal-security,now > 1.187.6~20.04.1+2.06-2ubuntu14.4 amd64 [installed] > 148c148 > < klibc-utils/focal-updates,focal-security,now 2.0.7-1ubuntu5.1 amd64 > [installed,automatic] > --- > > klibc-utils/focal-updates,focal-security,now 2.0.7-1ubuntu5.2 amd64 > [installed,automatic] > 152c152 > < landscape-common/focal-updates,now 19.12-0ubuntu4.3 amd64 > [installed,automatic] > --- > > landscape-common/now 19.12-0ubuntu4.3 amd64 [installed,upgradable to: > 23.02-0ubuntu1~20.04.2] > 154,155c154,155 > < less/now 551-1ubuntu0.1 amd64 [installed,upgradable to: 551-1ubuntu0.2] > < libaccountsservice0/now 0.6.55-0ubuntu12~20.04.5 amd64 > [installed,upgradable to: 0.6.55-0ubuntu12~20.04.7] > --- > > less/now 551-1ubuntu0.1 amd64 [installed,upgradable to: 551-1ubuntu0.3] > > libaccountsservice0/focal-updates,focal-security,now > 0.6.55-0ubuntu12~20.04.7 amd64 [installed,automatic] > 175c175 > < libblkid1/focal-security,now 2.34-0.1ubuntu9.3 amd64 > [installed,upgradable to: 2.34-0.1ubuntu9.4] > --- > > libblkid1/now 2.34-0.1ubuntu9.3 amd64 [installed,upgradable to: > 2.34-0.1ubuntu9.6] > 187,190c187,190 > < libc-bin/now 2.31-0ubuntu9.9 amd64 [installed,upgradable to: > 2.31-0ubuntu9.14] > < libc-dev-bin/focal-updates,focal-security,now 2.31-0ubuntu9.14 amd64 > [installed,automatic] > < libc6-dev/focal-updates,focal-security,now 2.31-0ubuntu9.14 amd64 > [installed,automatic] > < libc6/focal-updates,focal-security,now 2.31-0ubuntu9.14 amd64 > [installed,automatic] > --- > > libc-bin/now 2.31-0ubuntu9.9 amd64 [installed,upgradable to: > 2.31-0ubuntu9.15] > > libc-dev-bin/focal-updates,focal-security,now 2.31-0ubuntu9.15 amd64 > [installed,automatic] > > libc6-dev/focal-updates,focal-security,now 2.31-0ubuntu9.15 amd64 > [installed,automatic] > > libc6/focal-updates,focal-security,now 2.31-0ubuntu9.15 amd64 > [installed,automatic] > 203,204c203,204 > < libcurl3-gnutls/now 7.68.0-1ubuntu2.15 amd64 [installed,upgradable to: > 7.68.0-1ubuntu2.21] > < libcurl4/focal-updates,focal-security,now 7.68.0-1ubuntu2.21 amd64 > [installed,automatic] > --- > > libcurl3-gnutls/now 7.68.0-1ubuntu2.15 amd64 [installed,upgradable to: > 7.68.0-1ubuntu2.22] > > libcurl4/focal-updates,focal-security,now 7.68.0-1ubuntu2.22 amd64 > [installed,automatic] > 227c227 > < libfdisk1/focal-security,now 2.34-0.1ubuntu9.3 amd64 > [installed,upgradable to: 2.34-0.1ubuntu9.4] > --- > > libfdisk1/now 2.34-0.1ubuntu9.3 amd64 [installed,upgradable to: > 2.34-0.1ubuntu9.6] > 244,246c244,246 > < libglib2.0-0/now 2.64.6-1~ubuntu20.04.4 amd64 [installed,upgradable to: > 2.64.6-1~ubuntu20.04.6] > < libglib2.0-bin/now 2.64.6-1~ubuntu20.04.4 amd64 [installed,upgradable > to: 2.64.6-1~ubuntu20.04.6] > < libglib2.0-data/now 2.64.6-1~ubuntu20.04.4 all [installed,upgradable to: > 2.64.6-1~ubuntu20.04.6] > --- > > libglib2.0-0/now 2.64.6-1~ubuntu20.04.4 amd64 [installed,upgradable to: > 2.64.6-1~ubuntu20.04.7] > > libglib2.0-bin/now 2.64.6-1~ubuntu20.04.4 amd64 [installed,upgradable > to: 2.64.6-1~ubuntu20.04.7] > > libglib2.0-data/now 2.64.6-1~ubuntu20.04.4 all [installed,upgradable to: > 2.64.6-1~ubuntu20.04.7] > 248c248 > < libgnutls30/now 3.6.13-2ubuntu1.7 amd64 [installed,upgradable to: > 3.6.13-2ubuntu1.10] > --- > > libgnutls30/now 3.6.13-2ubuntu1.7 amd64 [installed,upgradable to: > 3.6.13-2ubuntu1.11] > 278c278 > < libklibc/focal-updates,focal-security,now 2.0.7-1ubuntu5.1 amd64 > [installed,automatic] > --- > > libklibc/focal-updates,focal-security,now 2.0.7-1ubuntu5.2 amd64 > [installed,automatic] > 284c284 > < libldap-2.4-2/now 2.4.49+dfsg-2ubuntu1.9 amd64 [installed,upgradable to: > 2.4.49+dfsg-2ubuntu1.10] > --- > > libldap-2.4-2/focal-updates,focal-security,now 2.4.49+dfsg-2ubuntu1.10 > amd64 [installed,automatic] > 301c301 > < libmount1/focal-security,now 2.34-0.1ubuntu9.3 amd64 > [installed,upgradable to: 2.34-0.1ubuntu9.4] > --- > > libmount1/now 2.34-0.1ubuntu9.3 amd64 [installed,upgradable to: > 2.34-0.1ubuntu9.6] > 306,307c306,307 > < libncurses6/focal,now 6.2-0ubuntu2 amd64 [installed,upgradable to: > 6.2-0ubuntu2.1] > < libncursesw6/focal,now 6.2-0ubuntu2 amd64 [installed,upgradable to: > 6.2-0ubuntu2.1] > --- > > libncurses6/focal-updates,focal-security,now 6.2-0ubuntu2.1 amd64 > [installed,automatic] > > libncursesw6/focal-updates,focal-security,now 6.2-0ubuntu2.1 amd64 > [installed,automatic] > 314c314 > < libnghttp2-14/focal,now 1.40.0-1build1 amd64 [installed,upgradable to: > 1.40.0-1ubuntu0.2] > --- > > libnghttp2-14/focal,now 1.40.0-1build1 amd64 [installed,upgradable to: > 1.40.0-1ubuntu0.3] > 316,318c316,318 > < libnspr4/focal,now 2:4.25-1 amd64 [installed,automatic] > < libnss-systemd/now 245.4-4ubuntu3.19 amd64 [installed,upgradable to: > 245.4-4ubuntu3.23] > < libnss3/now 2:3.49.1-1ubuntu1.8 amd64 [installed,upgradable to: > 2:3.49.1-1ubuntu1.9] > --- > > libnspr4/focal,now 2:4.25-1 amd64 [installed,upgradable to: > 2:4.35-0ubuntu0.20.04.1] > > libnss-systemd/focal-security,now 245.4-4ubuntu3.20 amd64 > [installed,upgradable to: 245.4-4ubuntu3.23] > > libnss3/now 2:3.49.1-1ubuntu1.8 amd64 [installed,upgradable to: > 2:3.98-0ubuntu0.20.04.2] > 325,328c325,328 > < libpam-modules-bin/now 1.3.1-5ubuntu4.6 amd64 [installed,upgradable to: > 1.3.1-5ubuntu4.7] > < libpam-modules/now 1.3.1-5ubuntu4.6 amd64 [installed,upgradable to: > 1.3.1-5ubuntu4.7] > < libpam-runtime/now 1.3.1-5ubuntu4.6 all [installed,upgradable to: > 1.3.1-5ubuntu4.7] > < libpam-systemd/now 245.4-4ubuntu3.19 amd64 [installed,upgradable to: > 245.4-4ubuntu3.23] > --- > > libpam-modules-bin/focal-updates,focal-security,now 1.3.1-5ubuntu4.7 > amd64 [installed,automatic] > > libpam-modules/focal-updates,focal-security,now 1.3.1-5ubuntu4.7 amd64 > [installed,automatic] > > libpam-runtime/focal-updates,focal-security,now 1.3.1-5ubuntu4.7 all > [installed,automatic] > > libpam-systemd/focal-security,now 245.4-4ubuntu3.20 amd64 > [installed,upgradable to: 245.4-4ubuntu3.23] > 347c347 > < libprocps8/now 2:3.3.16-1ubuntu2.3 amd64 [installed,upgradable to: > 2:3.3.16-1ubuntu2.4] > --- > > libprocps8/focal-updates,focal-security,now 2:3.3.16-1ubuntu2.4 amd64 > [installed,automatic] > 375c375 > < libsmartcols1/focal-security,now 2.34-0.1ubuntu9.3 amd64 > [installed,upgradable to: 2.34-0.1ubuntu9.4] > --- > > libsmartcols1/now 2.34-0.1ubuntu9.3 amd64 [installed,upgradable to: > 2.34-0.1ubuntu9.6] > 386c386 > < libsystemd0/now 245.4-4ubuntu3.19 amd64 [installed,upgradable to: > 245.4-4ubuntu3.23] > --- > > libsystemd0/focal-security,now 245.4-4ubuntu3.20 amd64 > [installed,upgradable to: 245.4-4ubuntu3.23] > 394c394 > < libtinfo6/focal,now 6.2-0ubuntu2 amd64 [installed,upgradable to: > 6.2-0ubuntu2.1] > --- > > libtinfo6/focal-updates,focal-security,now 6.2-0ubuntu2.1 amd64 > [installed,automatic] > 407c407 > < libuuid1/focal-security,now 2.34-0.1ubuntu9.3 amd64 > [installed,upgradable to: 2.34-0.1ubuntu9.4] > --- > > libuuid1/now 2.34-0.1ubuntu9.3 amd64 [installed,upgradable to: > 2.34-0.1ubuntu9.6] > 434c434 > < linux-azure/now 5.15.0.1033.40~20.04.23 amd64 [installed,upgradable to: > 5.15.0.1059.67~20.04.49] > --- > > linux-azure/now 5.15.0.1033.40~20.04.23 amd64 [installed,upgradable to: > 5.15.0.1064.73~20.04.1] > 438,439c438,439 > < linux-cloud-tools-azure/now 5.15.0.1033.40~20.04.23 amd64 > [installed,upgradable to: 5.15.0.1059.67~20.04.49] > < linux-cloud-tools-common/focal-updates,focal-security,now 5.4.0-139.156 > all [installed,upgradable to: 5.4.0-174.193] > --- > > linux-cloud-tools-azure/now 5.15.0.1033.40~20.04.23 amd64 > [installed,upgradable to: 5.15.0.1064.73~20.04.1] > > linux-cloud-tools-common/focal-updates,focal-security,now 5.4.0-182.202 > all [installed] > 441c441 > < linux-headers-azure/now 5.15.0.1033.40~20.04.23 amd64 > [installed,upgradable to: 5.15.0.1059.67~20.04.49] > --- > > linux-headers-azure/now 5.15.0.1033.40~20.04.23 amd64 > [installed,upgradable to: 5.15.0.1064.73~20.04.1] > 443,444c443,444 > < linux-image-azure/now 5.15.0.1033.40~20.04.23 amd64 > [installed,upgradable to: 5.15.0.1059.67~20.04.49] > < linux-libc-dev/focal-updates,focal-security,now 5.4.0-174.193 amd64 > [installed,automatic] > --- > > linux-image-azure/now 5.15.0.1033.40~20.04.23 amd64 > [installed,upgradable to: 5.15.0.1064.73~20.04.1] > > linux-libc-dev/focal-updates,focal-security,now 5.4.0-182.202 amd64 > [installed,automatic] > 447,449c447,449 > < linux-tools-azure/now 5.15.0.1033.40~20.04.23 amd64 > [installed,upgradable to: 5.15.0.1059.67~20.04.49] > < linux-tools-common/focal-updates,focal-security,now 5.4.0-139.156 all > [installed,upgradable to: 5.4.0-174.193] > < locales/now 2.31-0ubuntu9.9 all [installed,upgradable to: > 2.31-0ubuntu9.14] > --- > > linux-tools-azure/now 5.15.0.1033.40~20.04.23 amd64 > [installed,upgradable to: 5.15.0.1064.73~20.04.1] > > linux-tools-common/focal-updates,focal-security,now 5.4.0-139.156 all > [installed,upgradable to: 5.4.0-182.202] > > locales/now 2.31-0ubuntu9.9 all [installed,upgradable to: > 2.31-0ubuntu9.15] > 473c473 > < mount/focal-security,now 2.34-0.1ubuntu9.3 amd64 [installed,upgradable > to: 2.34-0.1ubuntu9.4] > --- > > mount/focal-updates,focal-security,now 2.34-0.1ubuntu9.6 amd64 > [installed,automatic] > 499c499 > < passwd/now 1:4.8.1-1ubuntu5.20.04.4 amd64 [installed,upgradable to: > 1:4.8.1-1ubuntu5.20.04.5] > --- > > passwd/focal-updates,focal-security,now 1:4.8.1-1ubuntu5.20.04.5 amd64 > [installed,automatic] > 555c555 > < python3-jinja2/focal,now 2.10.1-2 all [installed,upgradable to: > 2.10.1-2ubuntu0.2] > --- > > python3-jinja2/focal-updates,focal-security,now 2.10.1-2ubuntu0.2 all > [installed] > 592,594c592,594 > < python3-twisted-bin/now 18.9.0-11ubuntu0.20.04.2 amd64 > [installed,upgradable to: 18.9.0-11ubuntu0.20.04.3] > < python3-twisted/now 18.9.0-11ubuntu0.20.04.2 all [installed,upgradable > to: 18.9.0-11ubuntu0.20.04.3] > < python3-update-manager/now 1:20.04.10.11 all [installed,upgradable to: > 1:20.04.10.20] > --- > > python3-twisted-bin/focal-updates,focal-security,now > 18.9.0-11ubuntu0.20.04.3 amd64 [installed,automatic] > > python3-twisted/focal-updates,focal-security,now > 18.9.0-11ubuntu0.20.04.3 all [installed,automatic] > > python3-update-manager/now 1:20.04.10.11 all [installed,upgradable to: > 1:20.04.10.21] > 618c618 > < snapd/now 2.58+20.04 amd64 [installed,upgradable to: 2.58+20.04.1] > --- > > snapd/now 2.58+20.04 amd64 [installed,upgradable to: 2.62+20.04] > 625,627c625,627 > < sudo/now 1.8.31-1ubuntu1.4 amd64 [installed,upgradable to: > 1.8.31-1ubuntu1.5] > < systemd-sysv/now 245.4-4ubuntu3.19 amd64 [installed,upgradable to: > 245.4-4ubuntu3.23] > < systemd/now 245.4-4ubuntu3.19 amd64 [installed,upgradable to: > 245.4-4ubuntu3.23] > --- > > sudo/focal-updates,focal-security,now 1.8.31-1ubuntu1.5 amd64 > [installed,automatic] > > systemd-sysv/focal-security,now 245.4-4ubuntu3.20 amd64 > [installed,upgradable to: 245.4-4ubuntu3.23] > > systemd/focal-security,now 245.4-4ubuntu3.20 amd64 [installed,upgradable > to: 245.4-4ubuntu3.23] > 637c637 > < ubuntu-advantage-tools/now 27.13.5~20.04.1 amd64 [installed,upgradable > to: 31.2~20.04] > --- > > ubuntu-advantage-tools/now 27.13.5~20.04.1 amd64 [installed,upgradable > to: 31.2.3~20.04] > 648,649c648,649 > < update-manager-core/now 1:20.04.10.11 all [installed,upgradable to: > 1:20.04.10.20] > < update-notifier-common/now 3.192.30.16 all [installed,upgradable to: > 3.192.30.17] > --- > > update-manager-core/now 1:20.04.10.11 all [installed,upgradable to: > 1:20.04.10.21] > > update-notifier-common/now 3.192.30.16 all [installed,upgradable to: > 3.192.30.19] > 654,655c654,655 > < util-linux/focal-security,now 2.34-0.1ubuntu9.3 amd64 > [installed,upgradable to: 2.34-0.1ubuntu9.4] > < uuid-runtime/focal-security,now 2.34-0.1ubuntu9.3 amd64 > [installed,upgradable to: 2.34-0.1ubuntu9.4] > --- > > util-linux/now 2.34-0.1ubuntu9.3 amd64 [installed,upgradable to: > 2.34-0.1ubuntu9.6] > > uuid-runtime/now 2.34-0.1ubuntu9.3 amd64 [installed,upgradable to: > 2.34-0.1ubuntu9.6] > _______________________________________________ > varnish-misc mailing list > varnish-misc at varnish-cache.org > https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc > -------------- next part -------------- An HTML attachment was scrubbed... URL: From batanun at hotmail.com Mon May 20 08:32:05 2024 From: batanun at hotmail.com (Batanun B) Date: Mon, 20 May 2024 08:32:05 +0000 Subject: Varnish suddenly started using much more memory In-Reply-To: References: Message-ID: > Sorry, I should have been clearer, I meant: where are the varnish packages coming from? Are they from the official repositories, from https://packagecloud.io/varnishcache/ or built from source maybe? Ah, I see. They come from varnishcache packagecloud. More specifically, we use: https://packagecloud.io/install/repositories/varnishcache/varnish60lts/script.deb.sh > you should really invest some time in something like prometheus, it would probably have made the issue obvious Yes, in hindsight we definitely should have done that. I will discuss this with my coworkers going forward. > Is there any chance you can run the old version on the server to explore the differences? Possibly, for a limited time. If so, what types of tests would I do? And how long time would I need to run the old version? Note that with our setup, we wouldn't be able to run two different images at the same time, in the same environment, with both recieving traffic. So all traffic would be routed to this version (multiple servers, but all running the same image). An alternative approach that I'm considering, is to switch to the old image, but manually update the VCL to the new version. If the problem remains, then the issue is almost certainly with the VLC. But if the problem disapears, then it's more likely something else. > what's the output of: varnishstat -1 -f '*g_bytes' SMA.default.g_bytes ?10951750929 ? ? ? ? ?. ? Bytes outstanding SMA.large.g_bytes ? ? 8587329728 ? ? ? ? ?. ? Bytes outstanding SMA.Transient.g_bytes ? ? ?3177920 ? ? ? ? ?. ? Bytes outstanding So, the default storage usage has gone up with 2GB since my first message here, while the others have remained the same. Meanwhile, the total memory usage of Varnish has gone up to 26 GB, an increase of 3 GB. So now the overhead has gone up with 1GB to a total of 6 GB. Going forward, it will be interesting to see how the memory consumption changes after the default storage has reached its max (2 GB from where it is now). If we're lucky, it will stabilize, and then I'm not sure if it's worth it to troubleshoot any further. Otherwise, the free memory would get a bit too close to zero for our comfort, with no indication of stopping. Does Varnish keep track of total available OS memory, and start releasing memory by throwing out objects from the cache? Or will it continue to eat memory until something fails? > have you tweaked any workspaces/thread parameters? Nope. As I said, we haven't changed any OS or Varnish configuration. From uday.polu at indiamart.com Tue May 28 11:17:49 2024 From: uday.polu at indiamart.com (Uday Kumar) Date: Tue, 28 May 2024 16:47:49 +0530 Subject: Preventing Caching in Varnish Based on Backend Response Header Message-ID: Hello all, We need to prevent caching in Varnish based on a specific header from the backend. Could you please suggest the best approach to achieve this? *Thanks & Regards,* *Uday Kumar* -------------- next part -------------- An HTML attachment was scrubbed... URL: From guillaume.quintard at gmail.com Tue May 28 12:23:04 2024 From: guillaume.quintard at gmail.com (Guillaume Quintard) Date: Tue, 28 May 2024 05:23:04 -0700 Subject: Preventing Caching in Varnish Based on Backend Response Header In-Reply-To: References: Message-ID: Hi Uday, Sure, the classic practice will do nicely: sub vcl_backend_response { if (beresp.http.that-specific-header) { # TTL should match the time during which that header is unlikely to change # do NOT set it to 0s or less ( https://info.varnish-software.com/blog/hit-for-miss-and-why-a-null-ttl-is-bad-for-you ) set beresp.ttl = 2m; set beresp.uncacheable = true; return (deliver); } } The main trick here is beresp.uncacheable, you do not have to return immediately if you still have modifications/checks to do on that response. Would that work for you? -- Guillaume Quintard On Tue, May 28, 2024 at 4:55?AM Uday Kumar wrote: > Hello all, > > We need to prevent caching in Varnish based on a specific header from the > backend. > > Could you please suggest the best approach to achieve this? > > > *Thanks & Regards,* > *Uday Kumar* > _______________________________________________ > varnish-misc mailing list > varnish-misc at varnish-cache.org > https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc > -------------- next part -------------- An HTML attachment was scrubbed... URL: From uday.polu at indiamart.com Tue May 28 13:37:44 2024 From: uday.polu at indiamart.com (Uday Kumar) Date: Tue, 28 May 2024 19:07:44 +0530 Subject: Preventing Caching in Varnish Based on Backend Response Header In-Reply-To: References: Message-ID: Hello Guillaume, Great to know about this, it should work for us! will check this out *Thanks & Regards,* *Uday Kumar* On Tue, May 28, 2024 at 5:53?PM Guillaume Quintard < guillaume.quintard at gmail.com> wrote: > Hi Uday, > > Sure, the classic practice will do nicely: > > sub vcl_backend_response { > if (beresp.http.that-specific-header) { > # TTL should match the time during which that header is unlikely > to change > # do NOT set it to 0s or less ( > https://info.varnish-software.com/blog/hit-for-miss-and-why-a-null-ttl-is-bad-for-you > ) > set beresp.ttl = 2m; > set beresp.uncacheable = true; > return (deliver); > } > } > > The main trick here is beresp.uncacheable, you do not have to return > immediately if you still have modifications/checks to do on that response. > > Would that work for you? > > -- > Guillaume Quintard > > > On Tue, May 28, 2024 at 4:55?AM Uday Kumar > wrote: > >> Hello all, >> >> We need to prevent caching in Varnish based on a specific header from the >> backend. >> >> Could you please suggest the best approach to achieve this? >> >> >> *Thanks & Regards,* >> *Uday Kumar* >> _______________________________________________ >> varnish-misc mailing list >> varnish-misc at varnish-cache.org >> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: