From martynas at atomgraph.com Sat Oct 2 22:14:18 2021 From: martynas at atomgraph.com (=?UTF-8?Q?Martynas_Jusevi=C4=8Dius?=) Date: Sun, 3 Oct 2021 00:14:18 +0200 Subject: varnishadm exit code 200 Message-ID: Hi, We recently switched to the varnish:latest container and based an unprivileged image on it (entrypoint runs as USER varnish): https://github.com/AtomGraph/varnish/blob/official-image/Dockerfile We noticed that our varnishadm commands started failing. More specifically: root at dc17c642d39a:/etc/varnish# varnishadm "ban req.url ~ /" 200 As I understand it's a 200 exit code, which means varnishadm failed: https://varnish-cache.org/docs/5.1/reference/varnishadm.html#exit-status What does 200 mean exactly? I couldn't find any code list. My guess is that this has to do with the unprivileged varnish user, but I'm not sure what it takes to fix it. Martynas From martynas at atomgraph.com Sat Oct 2 22:23:33 2021 From: martynas at atomgraph.com (=?UTF-8?Q?Martynas_Jusevi=C4=8Dius?=) Date: Sun, 3 Oct 2021 00:23:33 +0200 Subject: varnishadm exit code 200 In-Reply-To: References: Message-ID: Actually it does not seem to be the exit code. I tried checking and it looks like the exit code is 0: root at dc17c642d39a:/etc/varnish# varnishadm "ban req.url ~ /" 200 root at dc17c642d39a:/etc/varnish# test $? -eq 0 || echo "Error" root at dc17c642d39a:/etc/varnish# So where is that "200" coming from? On Sun, Oct 3, 2021 at 12:14 AM Martynas Jusevi?ius wrote: > > Hi, > > We recently switched to the varnish:latest container and based an > unprivileged image on it (entrypoint runs as USER varnish): > https://github.com/AtomGraph/varnish/blob/official-image/Dockerfile > > We noticed that our varnishadm commands started failing. More specifically: > > root at dc17c642d39a:/etc/varnish# varnishadm "ban req.url ~ /" > 200 > > As I understand it's a 200 exit code, which means varnishadm failed: > https://varnish-cache.org/docs/5.1/reference/varnishadm.html#exit-status > > What does 200 mean exactly? I couldn't find any code list. > My guess is that this has to do with the unprivileged varnish user, > but I'm not sure what it takes to fix it. > > > Martynas From guillaume.quintard at gmail.com Sat Oct 2 22:39:07 2021 From: guillaume.quintard at gmail.com (Guillaume Quintard) Date: Sat, 2 Oct 2021 15:39:07 -0700 Subject: varnishadm exit code 200 In-Reply-To: References: Message-ID: 200 means the command passed to varnish, via varnishadm, succeeded. What makes you think it failed? On Sat, Oct 2, 2021, 15:24 Martynas Jusevi?ius wrote: > Actually it does not seem to be the exit code. I tried checking and it > looks like the exit code is 0: > > root at dc17c642d39a:/etc/varnish# varnishadm "ban req.url ~ /" > 200 > > root at dc17c642d39a:/etc/varnish# test $? -eq 0 || echo "Error" > root at dc17c642d39a:/etc/varnish# > > So where is that "200" coming from? > > On Sun, Oct 3, 2021 at 12:14 AM Martynas Jusevi?ius > wrote: > > > > Hi, > > > > We recently switched to the varnish:latest container and based an > > unprivileged image on it (entrypoint runs as USER varnish): > > https://github.com/AtomGraph/varnish/blob/official-image/Dockerfile > > > > We noticed that our varnishadm commands started failing. More > specifically: > > > > root at dc17c642d39a:/etc/varnish# varnishadm "ban req.url ~ /" > > 200 > > > > As I understand it's a 200 exit code, which means varnishadm failed: > > https://varnish-cache.org/docs/5.1/reference/varnishadm.html#exit-status > > > > What does 200 mean exactly? I couldn't find any code list. > > My guess is that this has to do with the unprivileged varnish user, > > but I'm not sure what it takes to fix it. > > > > > > Martynas > _______________________________________________ > varnish-misc mailing list > varnish-misc at varnish-cache.org > https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc > -------------- next part -------------- An HTML attachment was scrubbed... URL: From martynas at atomgraph.com Sat Oct 2 22:54:16 2021 From: martynas at atomgraph.com (=?UTF-8?Q?Martynas_Jusevi=C4=8Dius?=) Date: Sun, 3 Oct 2021 00:54:16 +0200 Subject: varnishadm exit code 200 In-Reply-To: References: Message-ID: I thought that was a non-zero exit code but it's not :) We just noticed a change -- in the earlier version we used (not sure which now, I think 4.x) there was no output from varnishadm. We'll just send the 200 to /dev/null then. On Sun, Oct 3, 2021 at 12:39 AM Guillaume Quintard wrote: > > 200 means the command passed to varnish, via varnishadm, succeeded. What makes you think it failed? > > On Sat, Oct 2, 2021, 15:24 Martynas Jusevi?ius wrote: >> >> Actually it does not seem to be the exit code. I tried checking and it >> looks like the exit code is 0: >> >> root at dc17c642d39a:/etc/varnish# varnishadm "ban req.url ~ /" >> 200 >> >> root at dc17c642d39a:/etc/varnish# test $? -eq 0 || echo "Error" >> root at dc17c642d39a:/etc/varnish# >> >> So where is that "200" coming from? >> >> On Sun, Oct 3, 2021 at 12:14 AM Martynas Jusevi?ius >> wrote: >> > >> > Hi, >> > >> > We recently switched to the varnish:latest container and based an >> > unprivileged image on it (entrypoint runs as USER varnish): >> > https://github.com/AtomGraph/varnish/blob/official-image/Dockerfile >> > >> > We noticed that our varnishadm commands started failing. More specifically: >> > >> > root at dc17c642d39a:/etc/varnish# varnishadm "ban req.url ~ /" >> > 200 >> > >> > As I understand it's a 200 exit code, which means varnishadm failed: >> > https://varnish-cache.org/docs/5.1/reference/varnishadm.html#exit-status >> > >> > What does 200 mean exactly? I couldn't find any code list. >> > My guess is that this has to do with the unprivileged varnish user, >> > but I'm not sure what it takes to fix it. >> > >> > >> > Martynas >> _______________________________________________ >> varnish-misc mailing list >> varnish-misc at varnish-cache.org >> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc From guillaume.quintard at gmail.com Sat Oct 2 23:15:01 2021 From: guillaume.quintard at gmail.com (Guillaume Quintard) Date: Sat, 2 Oct 2021 16:15:01 -0700 Subject: varnishadm exit code 200 In-Reply-To: References: Message-ID: Oooooh, 4.x is ancient, glad you made the upgrade! On Sat, Oct 2, 2021, 15:54 Martynas Jusevi?ius wrote: > I thought that was a non-zero exit code but it's not :) > > We just noticed a change -- in the earlier version we used (not sure > which now, I think 4.x) there was no output from varnishadm. > > We'll just send the 200 to /dev/null then. > > On Sun, Oct 3, 2021 at 12:39 AM Guillaume Quintard > wrote: > > > > 200 means the command passed to varnish, via varnishadm, succeeded. What > makes you think it failed? > > > > On Sat, Oct 2, 2021, 15:24 Martynas Jusevi?ius > wrote: > >> > >> Actually it does not seem to be the exit code. I tried checking and it > >> looks like the exit code is 0: > >> > >> root at dc17c642d39a:/etc/varnish# varnishadm "ban req.url ~ /" > >> 200 > >> > >> root at dc17c642d39a:/etc/varnish# test $? -eq 0 || echo "Error" > >> root at dc17c642d39a:/etc/varnish# > >> > >> So where is that "200" coming from? > >> > >> On Sun, Oct 3, 2021 at 12:14 AM Martynas Jusevi?ius > >> wrote: > >> > > >> > Hi, > >> > > >> > We recently switched to the varnish:latest container and based an > >> > unprivileged image on it (entrypoint runs as USER varnish): > >> > https://github.com/AtomGraph/varnish/blob/official-image/Dockerfile > >> > > >> > We noticed that our varnishadm commands started failing. More > specifically: > >> > > >> > root at dc17c642d39a:/etc/varnish# varnishadm "ban req.url ~ /" > >> > 200 > >> > > >> > As I understand it's a 200 exit code, which means varnishadm failed: > >> > > https://varnish-cache.org/docs/5.1/reference/varnishadm.html#exit-status > >> > > >> > What does 200 mean exactly? I couldn't find any code list. > >> > My guess is that this has to do with the unprivileged varnish user, > >> > but I'm not sure what it takes to fix it. > >> > > >> > > >> > Martynas > >> _______________________________________________ > >> varnish-misc mailing list > >> varnish-misc at varnish-cache.org > >> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc > -------------- next part -------------- An HTML attachment was scrubbed... URL: From leo.lelievre at protonmail.com Sun Oct 3 16:05:33 2021 From: leo.lelievre at protonmail.com (=?utf-8?B?TMOpbw==?=) Date: Sun, 03 Oct 2021 16:05:33 +0000 Subject: Varnish only resolve the ip on startup Message-ID: <5QFbmzVSZFr8Wa5TKmGoJPKlh_4tM5c5G1y8uaZJou9C2LWim5gJUdWz_eZX2xGiFbc7YQiwKS4viUeaRsABcgMYqwneIda5ynM-22evz-0=@protonmail.com> Hello, While setting up a docker-compose with a cache powered by varnish I came across an issue when recreating a container that varnish depends on. After the recreation varnish doesn't try to retrieve the new local ip. I've been told this is an upstream varnish issue (https://github.com/varnish/docker-varnish/issues/41). I have found some ways to solve the problem, but will an appropriate solution be considered ? Thanks in advance, Have a good day, L?o Lelievre -------------- next part -------------- An HTML attachment was scrubbed... URL: From dridi at varni.sh Mon Oct 4 03:52:20 2021 From: dridi at varni.sh (Dridi Boukelmoune) Date: Mon, 4 Oct 2021 03:52:20 +0000 Subject: varnishadm exit code 200 In-Reply-To: References: Message-ID: On Sat, Oct 2, 2021 at 10:24 PM Martynas Jusevi?ius wrote: > > Actually it does not seem to be the exit code. I tried checking and it > looks like the exit code is 0: > > root at dc17c642d39a:/etc/varnish# varnishadm "ban req.url ~ /" > 200 > > root at dc17c642d39a:/etc/varnish# test $? -eq 0 || echo "Error" > root at dc17c642d39a:/etc/varnish# > > So where is that "200" coming from? You shouldn't see a 200, except for this bug fixed in 7.0: https://github.com/varnishcache/varnish-cache/issues/3687 What version of Varnish are you running? Dridi From guillaume.quintard at gmail.com Mon Oct 4 15:43:19 2021 From: guillaume.quintard at gmail.com (Guillaume Quintard) Date: Mon, 4 Oct 2021 08:43:19 -0700 Subject: Varnish only resolve the ip on startup In-Reply-To: <5QFbmzVSZFr8Wa5TKmGoJPKlh_4tM5c5G1y8uaZJou9C2LWim5gJUdWz_eZX2xGiFbc7YQiwKS4viUeaRsABcgMYqwneIda5ynM-22evz-0=@protonmail.com> References: <5QFbmzVSZFr8Wa5TKmGoJPKlh_4tM5c5G1y8uaZJou9C2LWim5gJUdWz_eZX2xGiFbc7YQiwKS4viUeaRsABcgMYqwneIda5ynM-22evz-0=@protonmail.com> Message-ID: I think it makes sense for Varnish to natively support backends changing their IPs. I do get the performance argument but now that there is a cloud/container market and that Varnish has proven to be useful in it, this basic functionality should be brought in. Would it be acceptable to add a "host_string" to vrt_endpoint and fill it if the VCL backend isn't an IP, then, we can add another cp_methods to cache_conn_pool.c to use it? This way IPs are still super fast, and hostnames become actually useful and a bit less confusing? -- Guillaume Quintard On Sun, Oct 3, 2021 at 9:07 AM L?o wrote: > Hello, > While setting up a docker-compose with a cache powered by varnish I came > across an issue when recreating a container that varnish depends on. After > the recreation varnish doesn't try to retrieve the new local ip. I've been > told this is an upstream varnish issue ( > https://github.com/varnish/docker-varnish/issues/41). I have found some > ways to solve the problem, but will an appropriate solution be considered ? > > Thanks in advance, > Have a good day, > L?o Lelievre > _______________________________________________ > varnish-misc mailing list > varnish-misc at varnish-cache.org > https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc > -------------- next part -------------- An HTML attachment was scrubbed... URL: From dridi at varni.sh Mon Oct 4 16:48:36 2021 From: dridi at varni.sh (Dridi Boukelmoune) Date: Mon, 4 Oct 2021 16:48:36 +0000 Subject: Varnish only resolve the ip on startup In-Reply-To: References: <5QFbmzVSZFr8Wa5TKmGoJPKlh_4tM5c5G1y8uaZJou9C2LWim5gJUdWz_eZX2xGiFbc7YQiwKS4viUeaRsABcgMYqwneIda5ynM-22evz-0=@protonmail.com> Message-ID: On Mon, Oct 4, 2021 at 3:45 PM Guillaume Quintard wrote: > > I think it makes sense for Varnish to natively support backends changing their IPs. I do get the performance argument but now that there is a cloud/container market and that Varnish has proven to be useful in it, this basic functionality should be brought in. I would assume the primary argument was simplicity, not performance, but I wasn't around. One could argue it's turned into simplism in today's cloudy cloud cloudy world. > Would it be acceptable to add a "host_string" to vrt_endpoint and fill it if the VCL backend isn't an IP, then, we can add another cp_methods to cache_conn_pool.c to use it? This way IPs are still super fast, and hostnames become actually useful and a bit less confusing? One problem I have (and that you should be familiar with) is that portable interfaces we have that *respect* the system configuration (hosts file, nsswitch configuration etc) are not providing enough information. For example it becomes cumbersome to resolve SRV records or even get the TTL of individual records for a DNS resolution in a *portable* fashion. When you put it like this, it sounds simple enough (dare I say simplistic?) but what I see is a sizeable can of worms. I do think we could do something about it, I don't know what would be satisfying. Cheers, Dridi From guillaume.quintard at gmail.com Mon Oct 4 17:15:47 2021 From: guillaume.quintard at gmail.com (Guillaume Quintard) Date: Mon, 4 Oct 2021 10:15:47 -0700 Subject: Varnish only resolve the ip on startup In-Reply-To: References: <5QFbmzVSZFr8Wa5TKmGoJPKlh_4tM5c5G1y8uaZJou9C2LWim5gJUdWz_eZX2xGiFbc7YQiwKS4viUeaRsABcgMYqwneIda5ynM-22evz-0=@protonmail.com> Message-ID: On Mon, Oct 4, 2021 at 9:49 AM Dridi Boukelmoune wrote: > One problem I have (and that you should be familiar with) is that > portable interfaces we have that *respect* the system configuration > (hosts file, nsswitch configuration etc) are not providing enough > information. For example it becomes cumbersome to resolve SRV records > or even get the TTL of individual records for a DNS resolution in a > *portable* fashion. > > When you put it like this, it sounds simple enough (dare I say > simplistic?) but what I see is a sizeable can of worms. > That sounds like a bit of a strawman to me. getaddrinfo and connect are standard, and that's about all we should need. Applications are supposed (in general) to just use whatever gai give them. We can call them every time we need a new connection so we don't worry about TTL, and we just disregard SRV records. The vast majority of users don't need SRV (yet?), and don't expect the application to optimize DNS calls, but they do complain that giving a hostname to VCL doesn't work. Let's just provide basic, expected functionality out of the box, and leave the fancier features to vmod_goto and vmod_dynamic. -- Guillaume Quintard -------------- next part -------------- An HTML attachment was scrubbed... URL: From justinl at arena.net Mon Oct 4 17:29:55 2021 From: justinl at arena.net (Justin Lloyd) Date: Mon, 4 Oct 2021 17:29:55 +0000 Subject: Varnish only resolve the ip on startup In-Reply-To: References: <5QFbmzVSZFr8Wa5TKmGoJPKlh_4tM5c5G1y8uaZJou9C2LWim5gJUdWz_eZX2xGiFbc7YQiwKS4viUeaRsABcgMYqwneIda5ynM-22evz-0=@protonmail.com> Message-ID: I?m definitely watching this topic, considering I?m planning on moving to Varnish Enterprise next year and putting a cluster in ECS, if not Fargate, so being able to easily handle dynamic IPs would be extremely helpful. From: varnish-misc On Behalf Of Guillaume Quintard Sent: Monday, October 4, 2021 10:16 AM To: Dridi Boukelmoune Cc: varnish-misc at varnish-cache.org Subject: Re: Varnish only resolve the ip on startup On Mon, Oct 4, 2021 at 9:49 AM Dridi Boukelmoune > wrote: One problem I have (and that you should be familiar with) is that portable interfaces we have that *respect* the system configuration (hosts file, nsswitch configuration etc) are not providing enough information. For example it becomes cumbersome to resolve SRV records or even get the TTL of individual records for a DNS resolution in a *portable* fashion. When you put it like this, it sounds simple enough (dare I say simplistic?) but what I see is a sizeable can of worms. That sounds like a bit of a strawman to me. getaddrinfo and connect are standard, and that's about all we should need. Applications are supposed (in general) to just use whatever gai give them. We can call them every time we need a new connection so we don't worry about TTL, and we just disregard SRV records. The vast majority of users don't need SRV (yet?), and don't expect the application to optimize DNS calls, but they do complain that giving a hostname to VCL doesn't work. Let's just provide basic, expected functionality out of the box, and leave the fancier features to vmod_goto and vmod_dynamic. -- Guillaume Quintard -------------- next part -------------- An HTML attachment was scrubbed... URL: From dridi at varni.sh Mon Oct 4 18:38:11 2021 From: dridi at varni.sh (Dridi Boukelmoune) Date: Mon, 4 Oct 2021 18:38:11 +0000 Subject: Varnish only resolve the ip on startup In-Reply-To: References: <5QFbmzVSZFr8Wa5TKmGoJPKlh_4tM5c5G1y8uaZJou9C2LWim5gJUdWz_eZX2xGiFbc7YQiwKS4viUeaRsABcgMYqwneIda5ynM-22evz-0=@protonmail.com> Message-ID: On Mon, Oct 4, 2021 at 5:29 PM Justin Lloyd wrote: > > I?m definitely watching this topic, considering I?m planning on moving to Varnish Enterprise next year and putting a cluster in ECS, if not Fargate, so being able to easily handle dynamic IPs would be extremely helpful. Like I implied in my initial response you have this ability out of the box with Varnish Enterprise, and with Varnish Cache there are third-party VMODs. Dridi From dridi at varni.sh Mon Oct 4 18:54:56 2021 From: dridi at varni.sh (Dridi Boukelmoune) Date: Mon, 4 Oct 2021 18:54:56 +0000 Subject: Varnish only resolve the ip on startup In-Reply-To: References: <5QFbmzVSZFr8Wa5TKmGoJPKlh_4tM5c5G1y8uaZJou9C2LWim5gJUdWz_eZX2xGiFbc7YQiwKS4viUeaRsABcgMYqwneIda5ynM-22evz-0=@protonmail.com> Message-ID: On Mon, Oct 4, 2021 at 5:15 PM Guillaume Quintard wrote: > > On Mon, Oct 4, 2021 at 9:49 AM Dridi Boukelmoune wrote: >> >> One problem I have (and that you should be familiar with) is that >> portable interfaces we have that *respect* the system configuration >> (hosts file, nsswitch configuration etc) are not providing enough >> information. For example it becomes cumbersome to resolve SRV records >> or even get the TTL of individual records for a DNS resolution in a >> *portable* fashion. >> >> When you put it like this, it sounds simple enough (dare I say >> simplistic?) but what I see is a sizeable can of worms. > > > That sounds like a bit of a strawman to me. getaddrinfo and connect are standard, and that's about all we should need. Applications are supposed (in general) to just use whatever gai give them. We can call them every time we need a new connection so we don't worry about TTL, and we just disregard SRV records. > The vast majority of users don't need SRV (yet?), and don't expect the application to optimize DNS calls, but they do complain that giving a hostname to VCL doesn't work. The "portable" interface I was referring to includes getaddrinfo, and I guess your suggestion would be to always resolve and leave A/AAAA records caching to your stub and/or recursive resolver. Fair enough, simplicity. That still doesn't seal the can of worms: once there are more than one address per family or addresses change, it's our connection and pooling models that need to be revisited, how many different addresses to try connecting to, how to retry, and how we account for all of that (stats for example). Again, it's a bit more complex that just saying "change the connect callback to one that combines resolve+connect". I'm not against a holistic management of connections, there are actually more aspects that I think should be at the core of our connection management like certificates attached to a session (and in particular their domains) once we have TLS support. I just think it needs to be better defined and not give the impression that messing about a new set of backend callbacks that will systematically resolve endpoints is as simple as it gets. Cheers From guillaume.quintard at gmail.com Tue Oct 5 03:11:28 2021 From: guillaume.quintard at gmail.com (Guillaume Quintard) Date: Mon, 4 Oct 2021 20:11:28 -0700 Subject: Varnish only resolve the ip on startup In-Reply-To: References: <5QFbmzVSZFr8Wa5TKmGoJPKlh_4tM5c5G1y8uaZJou9C2LWim5gJUdWz_eZX2xGiFbc7YQiwKS4viUeaRsABcgMYqwneIda5ynM-22evz-0=@protonmail.com> Message-ID: On Mon, Oct 4, 2021 at 11:55 AM Dridi Boukelmoune wrote: > That still doesn't seal the can of worms: once there are more than one > address per family or addresses change, it's our connection and > pooling models that need to be revisited, how many different addresses > to try connecting to, how to retry, and how we account for all of that > (stats for example). Again, it's a bit more complex that just saying > "change the connect callback to one that combines resolve+connect". > I do understand that it's the core of the problem, and I'm probably being pig-headed on this, but it feels to me that it's not really different from a server with a floating IP, or a level-4 load-balancer fronting the backend. The addresses may change, there may be more than one, but once the connection is open, you can trust it and keep using it. We trust connect() to get us to our goal, using getaddrinfo() on top of it just means that we trust the DNS server to provide good info. So: - if we prefer IPv6, we go through the list and pick the first IPv6 entry, if we don't find one, we just grab the first IPv4 entry, no second chance - if you try to reuse a connection and it died on you, you try to resolve+connect a new one. Maybe you get the same IP, maybe you don't, but we trust the DNS implementation to shuffle the entries around - stats is an interesting issue, but again, only if you let it be. Backends are already "IP address" or "UDS path", I'm fine with hiding all the DNS entries behind "non-numerical host". I agree, things get gnarly when we start fiddling with DNs, but there's that portable interface that allows us to not have to. And if you want fancy stuff like stats per IP, and fun load-balancing options that are DNS-aware, there are vmods for that. For the sake of transparency, I must admit that I do have an issue with my proposal though: there's no timeout option for getaddrinfo, which sucks. -- Guillaume Quintard -------------- next part -------------- An HTML attachment was scrubbed... URL: From slink at schokola.de Tue Oct 5 13:38:49 2021 From: slink at schokola.de (Nils Goroll) Date: Tue, 5 Oct 2021 15:38:49 +0200 Subject: Varnish only resolve the ip on startup In-Reply-To: <5QFbmzVSZFr8Wa5TKmGoJPKlh_4tM5c5G1y8uaZJou9C2LWim5gJUdWz_eZX2xGiFbc7YQiwKS4viUeaRsABcgMYqwneIda5ynM-22evz-0=@protonmail.com> References: <5QFbmzVSZFr8Wa5TKmGoJPKlh_4tM5c5G1y8uaZJou9C2LWim5gJUdWz_eZX2xGiFbc7YQiwKS4viUeaRsABcgMYqwneIda5ynM-22evz-0=@protonmail.com> Message-ID: <059d0011-41a9-a82c-bd8c-1ba467c85d53@schokola.de> On 03.10.21 18:05, L?o wrote: > I have found some ways to solve the problem https://github.com/nigoroll/libvmod-dynamic exists to do just that From guillaume.quintard at gmail.com Tue Oct 5 17:25:24 2021 From: guillaume.quintard at gmail.com (Guillaume Quintard) Date: Tue, 5 Oct 2021 10:25:24 -0700 Subject: Varnish only resolve the ip on startup In-Reply-To: <059d0011-41a9-a82c-bd8c-1ba467c85d53@schokola.de> References: <5QFbmzVSZFr8Wa5TKmGoJPKlh_4tM5c5G1y8uaZJou9C2LWim5gJUdWz_eZX2xGiFbc7YQiwKS4viUeaRsABcgMYqwneIda5ynM-22evz-0=@protonmail.com> <059d0011-41a9-a82c-bd8c-1ba467c85d53@schokola.de> Message-ID: I'm fine with adding vmod_dynamic to the varnish docker images so that users have access to this basic functionality, but, for the record: - I really feel this should be considered a basic feature and exist in core - if the docker image starts adding downstream vmods, it opens the gates to a flood of "can you add this vmod too?" questions, which I'm not looking forward to. -- Guillaume Quintard On Tue, Oct 5, 2021 at 6:39 AM Nils Goroll wrote: > On 03.10.21 18:05, L?o wrote: > > I have found some ways to solve the problem > > https://github.com/nigoroll/libvmod-dynamic exists to do just that > > _______________________________________________ > varnish-misc mailing list > varnish-misc at varnish-cache.org > https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc > -------------- next part -------------- An HTML attachment was scrubbed... URL: From leo.lelievre at protonmail.com Tue Oct 5 17:47:18 2021 From: leo.lelievre at protonmail.com (=?utf-8?B?TMOpbw==?=) Date: Tue, 05 Oct 2021 17:47:18 +0000 Subject: Varnish only resolve the ip on startup In-Reply-To: <059d0011-41a9-a82c-bd8c-1ba467c85d53@schokola.de> References: <5QFbmzVSZFr8Wa5TKmGoJPKlh_4tM5c5G1y8uaZJou9C2LWim5gJUdWz_eZX2xGiFbc7YQiwKS4viUeaRsABcgMYqwneIda5ynM-22evz-0=@protonmail.com> <059d0011-41a9-a82c-bd8c-1ba467c85d53@schokola.de> Message-ID: mailto:slink at schokola.de > On 03.10.21 18:05, L?o wrote: >> I have found some ways to solve the problem > > https://github.com/nigoroll/libvmod-dynamic exists to do just that The point was to keep the docker compose as simple as possible without the need to install other services -------------- next part -------------- An HTML attachment was scrubbed... URL: From dridi at varni.sh Wed Oct 6 05:46:40 2021 From: dridi at varni.sh (Dridi Boukelmoune) Date: Wed, 6 Oct 2021 05:46:40 +0000 Subject: Difference between adjusting set req backend in vcl_recv and vcl_backend_fetch In-Reply-To: References: Message-ID: On Mon, Sep 6, 2021 at 2:22 PM Hamidreza Hosseini wrote: > > Hi, > This is part of my varnish configuration file for sharding: > > ``` > cat /etc/varnish/default.vcl > > vcl 4.1; > > import directors; > > > probe myprobe { > .request = > "HEAD /healthcheck.php HTTP/1.1" > "Connection: close" > "User-Agent: Varnish Health Probe"; > .timeout = 1s; > .interval = 5s; > .window = 5; > .threshold = 3; > } > > > backend b-01 { .host = "b-01"; .port = "80"; .probe = myprobe; } > backend b-02 { .host = "b-02"; .port = "80"; .probe = myprobe; } > backend b-03 { .host = "b-03"; .port = "80"; .probe = myprobe; } > > > sub vcl_init { > > > new hls_cluster = directors.shard(); > hls_cluster.add_backend(b-01); > hls_cluster.add_backend(b-02); > hls_cluster.add_backend(b-03); > > > new p = directors.shard_param(); > > hls_cluster.set_rampup(30s); > # hls_cluster.set_warmup(0.5); > > hls_cluster.reconfigure(); > hls_cluster.associate(p.use()); > > } > > > > acl purge { > "localhost"; > } > > > sub vcl_recv { > > set req.backend_hint = hls_cluster.backend(); > > } > > > sub vcl_backend_fetch { > > p.set(by=KEY, key=hls_cluster.key(bereq.url)); > set bereq.backend = hls_cluster.backend(resolve=LAZY, healthy=CHOSEN); > > } > > ``` > 1. there are two set backend in this config, one is on vcl_recv: > "set req.backend_hint = hls_cluster.backend();" > and one in vcl_backend_fetch: > "set bereq.backend = hls_cluster.backend(resolve=LAZY, healthy=CHOSEN);" > should I remove set in vcl_recv cause I think if I adjust it , all requset will go through this backend list and configs like healthy=CHOSEN in vcl_backend_fetch wouldnt be applied! Am I true? bereq.backend is initialized from req.backend_hint when a backend fetch is triggered. Setting bereq.backend will simply override anything one in vcl_recv. > 2.Actually what is difference between vcl_backend_fetch and vcl_recv? In vcl_recv you are processing a client request that was just received. In vcl_backend_fetch you are preparing a bereq (backend request) derived from req (client request) in before fetching the resource from the backend. https://varnish-cache.org/docs/6.0/users-guide/vcl-built-in-subs.html > 3.should I remove "set req.backend_hint" from vcl_recv? If the answer to "do I use the backend selection to make decisions in client subroutines?" is no, then you can remove it from vcl_recv. The shard director has task-specific settings, so some things you may configure in vcl_recv would not apply to vcl_backend_fetch, so if the answer to the question above was yes, you would probably need both. Best, Dridi From dridi at varni.sh Wed Oct 6 06:12:09 2021 From: dridi at varni.sh (Dridi Boukelmoune) Date: Wed, 6 Oct 2021 06:12:09 +0000 Subject: Best practice for caching scenario with different backend servers but same content In-Reply-To: References: Message-ID: On Mon, Aug 16, 2021 at 1:34 PM Hamidreza Hosseini wrote: > > > In that case, hashing the URL only would prevent you from adding new > domains through your Varnish server. It won't hurt if you know you > will only ever have one domain to deal with, but hashing the host will > also not hurt as long as you normalize it to a unique value. > > Hi, > Let me elaborate my architecture more: > I have some backend servers to serve hls fragments for video live stream,e.g: > > ``` > > hls_backend_01 > hls_backend_02 > hls_backend_03 > hls_backend_04 > hls_backend_05 > hls_backend_06 > hls_backend_07 > hls_backend_08 > hls_backend_09 > hls_backend_10 > > ``` > > There is same content on all hls backend servers, there are 5 varnish in front of them for caching > Now If I use round-robin director on Varnishes, because varnish would cache " req.http.host + req.url ", so for the same content but from different backends it would cache double! for example: > if varnish for the first request and "test.ts" file goes to "hls_backend_01" backend server, would cache it and > for the next request from other clients because it is using round-robin director > it goes to "hls_backend_02" and would cache the same file again due to different "req.http.host" > So now I have a solution to use Shard director based on "key=req.url" instead of round robin > another way is to use round robin but adjusting the hash vcl to something like bellow: > > ``` > > sub vcl_hash { > hash_data(req.url); > return (lookup); > } > > ``` > > In this way varnish just hash the "req.url" not "req.http.host" > So, Varnish would cache the content based on the content uniqueness not based on the difference between backends. > 1. At first, I asked how I can normalize it, Is it possible at all according to what I said!? > Would you please explain it more with an example? In this case I think you are confusing "req.http.host" (host header) with the backend host name. For example, if you reach one of your 5 Varnish servers via www.example.com that's what clients will use and that's what req.http.host will contain. Your backends FQDNs could be something like this: - hls01.internal.example.com - hls02.internal.example.com - hls03.internal.example.com - ... - hls10.internal.example.com As the example suggests, these domains should not be directly reached by clients if your goal is to proxy them with Varnish. Those internal FQDNs should have no effect on the cache key populated with hash_data(...). > 2. You give an example about other domains, In this case I do not understand what it has to do with the domain Let's say your clients can reach either example.com or www.example.com for the same service, or tomorrow you add more than your HLS service behind Varnish you may very well receive multiple host headers. > 3.Maybe I'm thinking in wrong way because if varnish hash the data based on req.url : 'hash_data(req.url)' It shouldn't cache the same content but different backends again! > for example my request is : In this case you are "hashing" the client request with hash_data(...) and it has nothing to do with backend selection. The fallback director will precisely not do any kind of traffic balancing since its purpose is to always select the first healthy backend in the insertion order. The shard director may rely on the request hash or other criteria as we already covered. > http://varnish-01:/hls/test.ts > for first request it goes to "hls_backend_01" backend and cache it and for next request it goes to "hls_backend_02" backend, > so for each request it caches it again because backends are different? All subsequent requests to http://varnish-01:/hls/test.ts should go to the same hls_backend_01 backend with the shard director. As long as there are no other criteria than the ones we already discussed. If you want consistency across all your Varnish servers, you should configure your shard directors identically, with the backends added in the same order (unlike your initial VCL example using the fallback director). Best, Dridi From justinl at arena.net Mon Oct 25 18:21:54 2021 From: justinl at arena.net (Justin Lloyd) Date: Mon, 25 Oct 2021 18:21:54 +0000 Subject: Query for authorization username Message-ID: In my dev environment, I have a few users configured to use Basic authorization (configured in the Nginx backend) and I'd like to be able to perform VSL queries based on the auth user. This is what I was able to come up with, but I'm wondering if there is a simpler way that I'm just not seeing. require blob; if (req.http.Authorization) { set req.http.X-Auth-User = regsub(blob.encode(IDENTITY, blob=blob.decode(BASE64, encoded=regsub(req.http.Authorization, "^Basic (.*)", "\1"))), ":.*$", ""); } varnishtop -I ReqHeader:X-Auth-User varnishlog -i ReqURL -q 'ReqHeader:X-Auth-User ~ "someuser"' Thanks, Justin -------------- next part -------------- An HTML attachment was scrubbed... URL: From guillaume.quintard at gmail.com Mon Oct 25 21:58:27 2021 From: guillaume.quintard at gmail.com (Guillaume Quintard) Date: Mon, 25 Oct 2021 14:58:27 -0700 Subject: Query for authorization username In-Reply-To: References: Message-ID: I think it's close to optimal, given the current tools. I would probably try to move away from regsub() and use vmod_str ( https://github.com/varnish/varnish-modules/blob/master/src/vmod_str.vcc#L42), and maaaaaaybe use multiple assignments rather than on big expressions, but that's a personal preference at this point. It would look like something like this in my mind (highly untested, don't sue me if your computer explodes): import var; import str; import blob; sub vcl_recv { if (str.split(req.http.Authorization, 0) == "Basic") { var.set("b64", str.split(req.http.Authorization, 1)); var.set("decoded", bob.transcode(encoding = BASE64URL, encoded = var.get("b64"))); set req.http.X-Auth-User = str.split(var.get("decoded"), 0, ":"); } } everything in one expression: set req.http.X-Auth-User = str.split( blob.transcode( encoding = BASE64URL, encoded = str.split(req.http.Authorization, 1)) ), 0, ":" ); You should possibly use blob.transcode() anyway. -- Guillaume Quintard On Mon, Oct 25, 2021 at 11:25 AM Justin Lloyd wrote: > In my dev environment, I have a few users configured to use Basic > authorization (configured in the Nginx backend) and I?d like to be able to > perform VSL queries based on the auth user. This is what I was able to come > up with, but I?m wondering if there is a simpler way that I?m just not > seeing. > > > > require blob; > > if (req.http.Authorization) { > > set req.http.X-Auth-User = regsub(blob.encode(IDENTITY, > > blob=blob.decode(BASE64, > > > encoded=regsub(req.http.Authorization, "^Basic (.*)", "\1"))), > > ":.*$", ""); > > } > > > > varnishtop -I ReqHeader:X-Auth-User > > varnishlog -i ReqURL -q 'ReqHeader:X-Auth-User ~ ?someuser?' > > > > Thanks, > > Justin > > > _______________________________________________ > varnish-misc mailing list > varnish-misc at varnish-cache.org > https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc > -------------- next part -------------- An HTML attachment was scrubbed... URL: From justinl at arena.net Tue Oct 26 19:01:32 2021 From: justinl at arena.net (Justin Lloyd) Date: Tue, 26 Oct 2021 19:01:32 +0000 Subject: Query for authorization username In-Reply-To: References: Message-ID: Hi Guillaume, Thanks for the feedback! I?m hesitant about adding the external modules just due to the additional complexity of building and maintaining them via my configuration management system (SaltStack) right now. Here?s what works for me by changing to blob.transcode() while still using regsub(): set req.http.X-Auth-User = regsub(blob.transcode( encoding=IDENTITY, decoding=BASE64, encoded=regsub(req.http.Authorization, "^Basic (.*)", "\1")), ":.*$", "" ); Best, Justin From: Guillaume Quintard Sent: Monday, October 25, 2021 2:58 PM To: Justin Lloyd Cc: varnish-misc at varnish-cache.org Subject: Re: Query for authorization username I think it's close to optimal, given the current tools. I would probably try to move away from regsub() and use vmod_str (https://github.com/varnish/varnish-modules/blob/master/src/vmod_str.vcc#L42), and maaaaaaybe use multiple assignments rather than on big expressions, but that's a personal preference at this point. It would look like something like this in my mind (highly untested, don't sue me if your computer explodes): import var; import str; import blob; sub vcl_recv { if (str.split(req.http.Authorization, 0) == "Basic") { var.set("b64", str.split(req.http.Authorization, 1)); var.set("decoded", bob.transcode(encoding = BASE64URL, encoded = var.get("b64"))); set req.http.X-Auth-User = str.split(var.get("decoded"), 0, ":"); } } everything in one expression: set req.http.X-Auth-User = str.split( blob.transcode( encoding = BASE64URL, encoded = str.split(req.http.Authorization, 1)) ), 0, ":" ); You should possibly use blob.transcode() anyway. -- Guillaume Quintard On Mon, Oct 25, 2021 at 11:25 AM Justin Lloyd > wrote: In my dev environment, I have a few users configured to use Basic authorization (configured in the Nginx backend) and I?d like to be able to perform VSL queries based on the auth user. This is what I was able to come up with, but I?m wondering if there is a simpler way that I?m just not seeing. require blob; if (req.http.Authorization) { set req.http.X-Auth-User = regsub(blob.encode(IDENTITY, blob=blob.decode(BASE64, encoded=regsub(req.http.Authorization, "^Basic (.*)", "\1"))), ":.*$", ""); } varnishtop -I ReqHeader:X-Auth-User varnishlog -i ReqURL -q 'ReqHeader:X-Auth-User ~ ?someuser?' Thanks, Justin _______________________________________________ varnish-misc mailing list varnish-misc at varnish-cache.org https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc -------------- next part -------------- An HTML attachment was scrubbed... URL: From Benny_Sham at macroview.com Wed Oct 27 06:32:09 2021 From: Benny_Sham at macroview.com (Benny Sham) Date: Wed, 27 Oct 2021 06:32:09 +0000 Subject: Empty varnishstat for Varnish 6.4.0 In-Reply-To: References: Message-ID: <98be0c9b7ee848648fcefddc3f1b10d9@macroview.com> Hi, I am trying to use "varnishstat -1", but it returns empty with exit code 0. However, varnishadm (as follows) is working. # varnishadm -T 127.0.0.1:6082 -S /etc/varnish/secret "help" Below is some information. Please let me know if more info is required. Thanks a lot. --- # varnishd -V varnishd (varnish-6.4.0 revision 13f137934ec1cf14af66baf7896311115ee35598) # apt-cache policy varnish varnish: Installed: 6.4.0-3 Candidate: 6.4.0-3 Version table: *** 6.4.0-3 500 100 http://archive.ubuntu.com/ubuntu groovy/universe amd64 Packages 100 /var/lib/dpkg/status # cat /etc/systemd/system/varnish.service ... ExecStart=/usr/sbin/varnishd -j unix,user=vcache -F -a :6081 -T localhost:6082 -f /etc/varnish/default.vcl -S /etc/varnish/secret -s default,256m ... Regards, Benny ________________________________ IMPORTANT NOTICE: This e-mail (and any attachments) is confidential and may not be distributed, copied or used without the prior consent of Macroview Telecom Ltd., its affiliates or related bodies (?the Macroview Group?). The contents of this email may as well be legally privileged or otherwise copyright-protected. If you are not the intended recipient please notify us immediately by reply e-mail or telephone and delete this email (and any attachment) from your system, and destroy any printed copy. Any view or opinion expressed in this email may be those of the sender and not the Macroview Group. Internet communications cannot be guaranteed to be timely, secure, error or virus-free. You should check for viruses before opening or using any attachments. We do not accept liability relating to computer virus, data corruption, delay, interruption, unauthorised access or unauthorised amendment. From Benny_Sham at macroview.com Wed Oct 27 08:31:16 2021 From: Benny_Sham at macroview.com (Benny Sham) Date: Wed, 27 Oct 2021 08:31:16 +0000 Subject: Empty varnishstat for Varnish 6.4.0 In-Reply-To: <98be0c9b7ee848648fcefddc3f1b10d9@macroview.com> References: <98be0c9b7ee848648fcefddc3f1b10d9@macroview.com> Message-ID: <65289a11bc6b442a96ac4df7c420b86c@macroview.com> Thanks to bjorn and simon in IRC, It is caused by version mismatch: varnishstat -> 6.2.1 varnishd -> 6.4.0 The problem is resolved. Regards, Benny -----Original Message----- From: Benny Sham Sent: Wednesday, October 27, 2021 2:32 PM To: 'varnish-misc at varnish-cache.org' Subject: Empty varnishstat for Varnish 6.4.0 Hi, I am trying to use "varnishstat -1", but it returns empty with exit code 0. However, varnishadm (as follows) is working. # varnishadm -T 127.0.0.1:6082 -S /etc/varnish/secret "help" Below is some information. Please let me know if more info is required. Thanks a lot. --- # varnishd -V varnishd (varnish-6.4.0 revision 13f137934ec1cf14af66baf7896311115ee35598) # apt-cache policy varnish varnish: Installed: 6.4.0-3 Candidate: 6.4.0-3 Version table: *** 6.4.0-3 500 100 http://archive.ubuntu.com/ubuntu groovy/universe amd64 Packages 100 /var/lib/dpkg/status # cat /etc/systemd/system/varnish.service ... ExecStart=/usr/sbin/varnishd -j unix,user=vcache -F -a :6081 -T localhost:6082 -f /etc/varnish/default.vcl -S /etc/varnish/secret -s default,256m ... Regards, Benny ________________________________ IMPORTANT NOTICE: This e-mail (and any attachments) is confidential and may not be distributed, copied or used without the prior consent of Macroview Telecom Ltd., its affiliates or related bodies (?the Macroview Group?). The contents of this email may as well be legally privileged or otherwise copyright-protected. If you are not the intended recipient please notify us immediately by reply e-mail or telephone and delete this email (and any attachment) from your system, and destroy any printed copy. Any view or opinion expressed in this email may be those of the sender and not the Macroview Group. Internet communications cannot be guaranteed to be timely, secure, error or virus-free. You should check for viruses before opening or using any attachments. We do not accept liability relating to computer virus, data corruption, delay, interruption, unauthorised access or unauthorised amendment.