From batanun at hotmail.com Fri Apr 14 11:51:06 2023 From: batanun at hotmail.com (Batanun B) Date: Fri, 14 Apr 2023 11:51:06 +0000 Subject: Possible to disable/inactivate a backend using VCL? Message-ID: Hi, We are currently working on a new feature that won't go live for several months still. This new feature has it's own backend in Varnish. Most of our VCL code is identical for all environments, and this code refers to the new backend, so it needs to be defined otherwise Varnish won't start. But in production we don't want to show anything of this feature. And we would like to have this backend completely disabled or inactivated in Varnish. Can we do that using VCL? Like forcing the health to be sick, or something similar. We would prefer to keep this inside the backend declaration if possible, and we would also prefer somethink not too "hackish" (like pointing it to a dead IP). Does Varnish has any recommended approach for this? I know there is a cli command to set the health, but as I said we would really prefer doing this using VCL only. -------------- next part -------------- An HTML attachment was scrubbed... URL: From guillaume.quintard at gmail.com Fri Apr 14 15:52:41 2023 From: guillaume.quintard at gmail.com (Guillaume Quintard) Date: Fri, 14 Apr 2023 08:52:41 -0700 Subject: Possible to disable/inactivate a backend using VCL? In-Reply-To: References: Message-ID: Hi Batanun, I'm curious, if it's completely deactivated what's the benefit of having it in the vcl? To answer your question: cheat! Add this somewhere in your vcl: sub vcl_recv { # the if statement will never work, but the backend is referenced, so the compiler won't bother you if (false) { set req.backend_hint = you_deactivated_backend; } } You could also start varnish with `-p vcc_feature=-err_unref` but I don't recommend it as it'll affect all the unreferenced symbols, even the ones you may care about. Hope that helps -- Guillaume Quintard On Fri, Apr 14, 2023 at 4:53?AM Batanun B wrote: > Hi, > > We are currently working on a new feature that won't go live for several > months still. This new feature has it's own backend in Varnish. Most of our > VCL code is identical for all environments, and this code refers to the new > backend, so it needs to be defined otherwise Varnish won't start. But in > production we don't want to show anything of this feature. And we would > like to have this backend completely disabled or inactivated in Varnish. > Can we do that using VCL? Like forcing the health to be sick, or something > similar. We would prefer to keep this inside the backend declaration if > possible, and we would also prefer somethink not too "hackish" (like > pointing it to a dead IP). > > Does Varnish has any recommended approach for this? I know there is a cli > command to set the health, but as I said we would really prefer doing this > using VCL only. > _______________________________________________ > varnish-misc mailing list > varnish-misc at varnish-cache.org > https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc > -------------- next part -------------- An HTML attachment was scrubbed... URL: From izghitu at gmail.com Mon Apr 17 08:21:17 2023 From: izghitu at gmail.com (George) Date: Mon, 17 Apr 2023 11:21:17 +0300 Subject: Strange Broken Pipe error from Varnish health checks Message-ID: Hi, I have a Varnish/nginx cluster running with varnish-7.1.2-1.el7.x86_64 on CentOS 7. The issue I am having comes from the varnish health checks. I am getting a "broken pipe" error in the nginx error log at random times like below: Apr 10 17:32:46 VARNISH-MASTER nginx_varnish_error: 2023/04/10 17:32:46 [info] 17808#17808: *67626636 writev() failed (32: Broken pipe), client: unix:, server: _, request: "GET /varnish_check HTTP/1.1", host: "0.0.0.0" The strange thing is that this error appears only when Varnish performs the health checks. I have other scripts doing it(nagios, curl, wget, AWS ELB) but those do not show any errors. In addition to this Varnish and nginx where the health checks occur are on the same server and it makes no difference if I use a TCP connection or socket based one. Below are the varnish vcl and nginx locations for the health checks: backend nginx_varnish { .path = "/run/nginx/nginx.sock"; .first_byte_timeout = 600s; .probe = health; } location = /varnish_check { keepalive_timeout 305; return 200 'Varnish Check'; access_log /var/log/nginx/varnish_check.log main; error_log /var/log/nginx/varnish_check_errors.log debug; error_log syslog:server=unix:/run/nginx_log.in.sock,facility=local1,tag=nginx_varnish_error,nohostname info; } Are there any docs I can read about how exactly varnish performs the health checks and what internal processes are involved? Did anyone happen to have similar issues? This is not causing any operational problems for the cluster but it is just something that I want to determine why it is happening because it just should not be happening. Please help THanks in advance. -------------- next part -------------- An HTML attachment was scrubbed... URL: From guillaume.quintard at gmail.com Mon Apr 17 14:31:48 2023 From: guillaume.quintard at gmail.com (Guillaume Quintard) Date: Mon, 17 Apr 2023 07:31:48 -0700 Subject: Strange Broken Pipe error from Varnish health checks In-Reply-To: References: Message-ID: Hi George, Just to be sure, how is the probe "health" defined in your VCL? Cheers, -- Guillaume Quintard On Mon, Apr 17, 2023 at 1:23?AM George wrote: > Hi, > > I have a Varnish/nginx cluster running with varnish-7.1.2-1.el7.x86_64 on > CentOS 7. > > The issue I am having comes from the varnish health checks. I am getting a > "broken pipe" error in the nginx error log at random times like below: > Apr 10 17:32:46 VARNISH-MASTER nginx_varnish_error: 2023/04/10 17:32:46 > [info] 17808#17808: *67626636 writev() failed (32: Broken pipe), client: > unix:, server: _, request: "GET /varnish_check HTTP/1.1", host: "0.0.0.0" > > The strange thing is that this error appears only when Varnish performs > the health checks. I have other scripts doing it(nagios, curl, wget, AWS > ELB) but those do not show any errors. In addition to this Varnish and > nginx where the health checks occur are on the same server and it makes no > difference if I use a TCP connection or socket based one. > > Below are the varnish vcl and nginx locations for the health checks: > backend nginx_varnish { > .path = "/run/nginx/nginx.sock"; > .first_byte_timeout = 600s; > .probe = health; > } > > location = /varnish_check { > keepalive_timeout 305; > return 200 'Varnish Check'; > access_log /var/log/nginx/varnish_check.log main; > error_log /var/log/nginx/varnish_check_errors.log debug; > error_log > syslog:server=unix:/run/nginx_log.in.sock,facility=local1,tag=nginx_varnish_error,nohostname > info; > } > > Are there any docs I can read about how exactly varnish performs the > health checks and what internal processes are involved? > Did anyone happen to have similar issues? This is not causing any > operational problems for the cluster but it is just something that I want > to determine why it is happening because it just should not be happening. > > Please help > THanks in advance. > > _______________________________________________ > varnish-misc mailing list > varnish-misc at varnish-cache.org > https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc > -------------- next part -------------- An HTML attachment was scrubbed... URL: From izghitu at gmail.com Mon Apr 17 14:47:20 2023 From: izghitu at gmail.com (George) Date: Mon, 17 Apr 2023 17:47:20 +0300 Subject: Strange Broken Pipe error from Varnish health checks In-Reply-To: References: Message-ID: Hi, Below is the probe "health", I forgot to send it the first time: probe health { .url = "/varnish_check"; .timeout = 2s; .interval = 5s; .window = 3; .threshold = 2; } Thanks ?n lun., 17 apr. 2023 la 17:32, Guillaume Quintard < guillaume.quintard at gmail.com> a scris: > Hi George, > > Just to be sure, how is the probe "health" defined in your VCL? > > Cheers, > > -- > Guillaume Quintard > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From guillaume.quintard at gmail.com Mon Apr 17 15:46:56 2023 From: guillaume.quintard at gmail.com (Guillaume Quintard) Date: Mon, 17 Apr 2023 08:46:56 -0700 Subject: Strange Broken Pipe error from Varnish health checks In-Reply-To: References: Message-ID: Thanks, I looked at the code quickly and I'd venture that maybe the /varnish_check is a bit too large and doesn't fit Varnish's probe buffer (we only care about the status line of the response anyway), so Varnish closes the connection while nginx isn't done yet. If it's that, it's not very polite, but it's harmless. -- Guillaume Quintard On Mon, Apr 17, 2023 at 7:49?AM George wrote: > Hi, > > Below is the probe "health", I forgot to send it the first time: > probe health { > .url = "/varnish_check"; > .timeout = 2s; > .interval = 5s; > .window = 3; > .threshold = 2; > } > > Thanks > > ?n lun., 17 apr. 2023 la 17:32, Guillaume Quintard < > guillaume.quintard at gmail.com> a scris: > >> Hi George, >> >> Just to be sure, how is the probe "health" defined in your VCL? >> >> Cheers, >> >> -- >> Guillaume Quintard >> >> _______________________________________________ > varnish-misc mailing list > varnish-misc at varnish-cache.org > https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc > -------------- next part -------------- An HTML attachment was scrubbed... URL: From izghitu at gmail.com Mon Apr 17 16:10:56 2023 From: izghitu at gmail.com (George) Date: Mon, 17 Apr 2023 19:10:56 +0300 Subject: Strange Broken Pipe error from Varnish health checks In-Reply-To: References: Message-ID: Hi, In our case the response body is few bytes (13), header+body is 170 bytes. Can this be a bug related to something else perhaps? Please let me know. Thanks ?n lun., 17 apr. 2023 la 18:47, Guillaume Quintard < guillaume.quintard at gmail.com> a scris: > Thanks, I looked at the code quickly and I'd venture that maybe the > /varnish_check is a bit too large and doesn't fit Varnish's probe buffer > (we only care about the status line of the response anyway), so Varnish > closes the connection while nginx isn't done yet. > > If it's that, it's not very polite, but it's harmless. > -- > Guillaume Quintard > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From guillaume.quintard at gmail.com Mon Apr 17 16:15:00 2023 From: guillaume.quintard at gmail.com (Guillaume Quintard) Date: Mon, 17 Apr 2023 09:15:00 -0700 Subject: Strange Broken Pipe error from Varnish health checks In-Reply-To: References: Message-ID: That code hasn't moved in a while, so I'd be surprised to see a bug there, but that's always possible. Any chance you could get a tcpdump of a probe request (from connection to disconnection) so we can see what's going on? -- Guillaume Quintard On Mon, Apr 17, 2023 at 9:12?AM George wrote: > Hi, > > In our case the response body is few bytes (13), header+body is 170 bytes. > Can this be a bug related to something else perhaps? > > Please let me know. > Thanks > > ?n lun., 17 apr. 2023 la 18:47, Guillaume Quintard < > guillaume.quintard at gmail.com> a scris: > >> Thanks, I looked at the code quickly and I'd venture that maybe the >> /varnish_check is a bit too large and doesn't fit Varnish's probe buffer >> (we only care about the status line of the response anyway), so Varnish >> closes the connection while nginx isn't done yet. >> >> If it's that, it's not very polite, but it's harmless. >> -- >> Guillaume Quintard >> >> _______________________________________________ > varnish-misc mailing list > varnish-misc at varnish-cache.org > https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc > -------------- next part -------------- An HTML attachment was scrubbed... URL: From izghitu at gmail.com Mon Apr 17 19:50:16 2023 From: izghitu at gmail.com (George) Date: Mon, 17 Apr 2023 22:50:16 +0300 Subject: Strange Broken Pipe error from Varnish health checks In-Reply-To: References: Message-ID: I am not sure how a tcpdump will help in any way. Since all the health check requests are virtually the same. Since the varnish thread that probes the backend dies every once in a while then it will never show anything with tcpdump or I virtually have no way to know when it will die in order to capture an incomplete request using tcpdump. Please advise. ?n lun., 17 apr. 2023 la 19:15, Guillaume Quintard < guillaume.quintard at gmail.com> a scris: > That code hasn't moved in a while, so I'd be surprised to see a bug there, > but that's always possible. > Any chance you could get a tcpdump of a probe request (from connection to > disconnection) so we can see what's going on? > -- > Guillaume Quintard > > > On Mon, Apr 17, 2023 at 9:12?AM George wrote: > >> Hi, >> >> In our case the response body is few bytes (13), header+body is 170 >> bytes. Can this be a bug related to something else perhaps? >> >> Please let me know. >> Thanks >> >> ?n lun., 17 apr. 2023 la 18:47, Guillaume Quintard < >> guillaume.quintard at gmail.com> a scris: >> >>> Thanks, I looked at the code quickly and I'd venture that maybe the >>> /varnish_check is a bit too large and doesn't fit Varnish's probe buffer >>> (we only care about the status line of the response anyway), so Varnish >>> closes the connection while nginx isn't done yet. >>> >>> If it's that, it's not very polite, but it's harmless. >>> -- >>> Guillaume Quintard >>> >>> _______________________________________________ >> varnish-misc mailing list >> varnish-misc at varnish-cache.org >> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From dridi at varni.sh Tue Apr 18 10:23:01 2023 From: dridi at varni.sh (Dridi Boukelmoune) Date: Tue, 18 Apr 2023 10:23:01 +0000 Subject: Strange Broken Pipe error from Varnish health checks In-Reply-To: References: Message-ID: On Mon, Apr 17, 2023 at 4:17?PM Guillaume Quintard wrote: > > That code hasn't moved in a while, so I'd be surprised to see a bug there, but that's always possible. > Any chance you could get a tcpdump of a probe request (from connection to disconnection) so we can see what's going on? It has: https://github.com/varnishcache/varnish-cache/pull/3886 But it shouldn't have changed the default behavior. Dridi From walid.boudebouda at varnish-software.com Tue Apr 18 10:41:45 2023 From: walid.boudebouda at varnish-software.com (walid boudebouda) Date: Tue, 18 Apr 2023 12:41:45 +0200 Subject: Strange Broken Pipe error from Varnish health checks In-Reply-To: References: Message-ID: > > > It has: https://github.com/varnishcache/varnish-cache/pull/3886 > This was introduced in 7.3 and he is running 7.1 Walid On Tue, Apr 18, 2023 at 12:24?PM Dridi Boukelmoune wrote: > On Mon, Apr 17, 2023 at 4:17?PM Guillaume Quintard > wrote: > > > > That code hasn't moved in a while, so I'd be surprised to see a bug > there, but that's always possible. > > Any chance you could get a tcpdump of a probe request (from connection > to disconnection) so we can see what's going on? > > It has: https://github.com/varnishcache/varnish-cache/pull/3886 > > But it shouldn't have changed the default behavior. > > Dridi > _______________________________________________ > varnish-misc mailing list > varnish-misc at varnish-cache.org > https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc > -------------- next part -------------- An HTML attachment was scrubbed... URL: From kokoniimasu at gmail.com Tue Apr 18 15:03:22 2023 From: kokoniimasu at gmail.com (kokoniimasu) Date: Wed, 19 Apr 2023 00:03:22 +0900 Subject: Improved github issue templates Message-ID: Hi, Varnish already have a issue template, but sometimes I see people ignore it and post questions, etc. (they are closed right away) I saw a post on Twitter about the great categorization of issues in ImageMagick. https://twitter.com/yoya/status/1644160988600733696 I thought this was very nice and made a sample to see if it could be done in Varnish as well. I think that by putting in a category, we can guide them to the right place. https://github.com/xcir/ghsandbox/issues (Feel free to create issues for testing.) When a new issue is made, the category will be displayed. Bug report required each information (Expected Behavior, Current Behavior...) Of course, I understand that it is also used as a TODO, so I also allow a free format description with "Don't see your issue here? Open a blank issue." at the bottom of the issue creation screen A warning is also included here to prevent unintended use. https://github.com/xcir/ghsandbox/issues/new I am thinking of submitting a PR if this might be useful, what do you think? config is here. https://github.com/xcir/ghsandbox/tree/main/.github/ISSUE_TEMPLATE I'd be happy to be helpful. Regards, -- Shohei Tanaka(@xcir) https://blog.xcir.net/ (JP) -------------- next part -------------- An HTML attachment was scrubbed... URL: From dridi at varni.sh Tue Apr 18 16:11:05 2023 From: dridi at varni.sh (Dridi Boukelmoune) Date: Tue, 18 Apr 2023 16:11:05 +0000 Subject: Improved github issue templates In-Reply-To: References: Message-ID: On Tue, Apr 18, 2023 at 3:05?PM kokoniimasu wrote: > > Hi, > > Varnish already have a issue template, but sometimes I see people ignore it and post questions, etc. (they are closed right away) > > I saw a post on Twitter about the great categorization of issues in ImageMagick. > > https://twitter.com/yoya/status/1644160988600733696 > > I thought this was very nice and made a sample to see if it could be done in Varnish as well. > I think that by putting in a category, we can guide them to the right place. > > https://github.com/xcir/ghsandbox/issues > (Feel free to create issues for testing.) > > When a new issue is made, the category will be displayed. > Bug report required each information (Expected Behavior, Current Behavior...) > > Of course, I understand that it is also used as a TODO, so I also allow a free format description with "Don't see your issue here? Open a blank issue." at the bottom of the issue creation screen If we can help it, I'd rather not have the blank issue link. > A warning is also included here to prevent unintended use. > > https://github.com/xcir/ghsandbox/issues/new > > I am thinking of submitting a PR if this might be useful, what do you think? YES PLEASE! I have had this deep in my backlog for a long time now, and never got around to fishing it out. I didn't even know that issue templates could include external links. > config is here. > https://github.com/xcir/ghsandbox/tree/main/.github/ISSUE_TEMPLATE > > I'd be happy to be helpful. Feel free to refresh the current template. Maybe we don't need as many markdown comments as we currently have today with the categorization you bring upfront. Many thanks! Dridi From izghitu at gmail.com Tue Apr 18 16:41:54 2023 From: izghitu at gmail.com (George) Date: Tue, 18 Apr 2023 19:41:54 +0300 Subject: Strange Broken Pipe error from Varnish health checks In-Reply-To: References: Message-ID: Hi, Attached is the packet capture for the health check Please check and advise. ?n lun., 17 apr. 2023 la 19:15, Guillaume Quintard < guillaume.quintard at gmail.com> a scris: > That code hasn't moved in a while, so I'd be surprised to see a bug there, > but that's always possible. > Any chance you could get a tcpdump of a probe request (from connection to > disconnection) so we can see what's going on? > -- > Guillaume Quintard > > -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: capture.pcap Type: application/octet-stream Size: 1446 bytes Desc: not available URL: From guillaume.quintard at gmail.com Tue Apr 18 18:26:34 2023 From: guillaume.quintard at gmail.com (Guillaume Quintard) Date: Tue, 18 Apr 2023 11:26:34 -0700 Subject: Strange Broken Pipe error from Varnish health checks In-Reply-To: References: Message-ID: Hi George, That pcap only contains HTTP info, it would be super useful to have the TCP packets, (SYN/ACK/FIN) to see who closes the connection on whom. -- Guillaume Quintard On Tue, Apr 18, 2023 at 9:43?AM George wrote: > Hi, > > Attached is the packet capture for the health check > > Please check and advise. > > > ?n lun., 17 apr. 2023 la 19:15, Guillaume Quintard < > guillaume.quintard at gmail.com> a scris: > >> That code hasn't moved in a while, so I'd be surprised to see a bug >> there, but that's always possible. >> Any chance you could get a tcpdump of a probe request (from connection to >> disconnection) so we can see what's going on? >> -- >> Guillaume Quintard >> >> _______________________________________________ > varnish-misc mailing list > varnish-misc at varnish-cache.org > https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc > -------------- next part -------------- An HTML attachment was scrubbed... URL: From batanun at hotmail.com Wed Apr 19 08:34:21 2023 From: batanun at hotmail.com (Batanun B) Date: Wed, 19 Apr 2023 08:34:21 +0000 Subject: Possible to disable/inactivate a backend using VCL? In-Reply-To: References: Message-ID: Hi Guillaume, > I'm curious, if it's completely deactivated what's the benefit of having it in the vcl? It is only intended to be deactivated in production (until we go live). Our test and staging environments have the backend active. > if (false) { > ??set req.backend_hint = you_deactivated_backend; > } Thanks, I will test this. My current prod-specific setup for this backend looks like this: backend theBackend { .host = "localhost"; .port = "9999"; .probe = { .interval = 1h; } } This seems to be working when testing it locally. It also solves the problem of having to assign some arbitrary ip or hostname (the actual backend host for this service hasn't been created in production yet, since we are several months away from go live), which actually was our main problem. What do you think about this approach instead? Preferably this would be a built in feature in Varnish, with a setting "disabled = true" or similar in the backend definition, and then it would not require any host or ip to be configured. -------------- next part -------------- An HTML attachment was scrubbed... URL: From batanun at hotmail.com Wed Apr 19 08:48:02 2023 From: batanun at hotmail.com (Batanun B) Date: Wed, 19 Apr 2023 08:48:02 +0000 Subject: Varnish won't start because backend host resolves to too many addresses, but they are all identical IPs Message-ID: All of the sudden Varnish fails to start in my development environment, and gives me the following error message: Message from VCC-compiler: Backend host "redacted-hostname": resolves to too many addresses. Only one IPv4 and one IPv6 are allowed. Please specify which exact address you want to use, we found all of these: 555.123.123.3:80 555.123.123.3:80 555.123.123.3:80 I have changed the hostname and the IP above to not expose our server, but all three IP numbers are 100% identical. Shouldn't Varnish be able to figure out that in that case it can just choose any one and it will work as expected? It really should remove duplicates, and only if there are more than one non-duplicate IP then it should fail. The problem is that the backend host is a so called "app service" in Microsoft Azure, which is basically a platform as a service (PaaS), where Microsoft handles the networking including the domain name (no user access it directly). I have no idea why it suddenly resolves to multiple duplicate IPs. -------------- next part -------------- An HTML attachment was scrubbed... URL: From batanun at hotmail.com Wed Apr 19 09:26:36 2023 From: batanun at hotmail.com (Batanun B) Date: Wed, 19 Apr 2023 09:26:36 +0000 Subject: Strange Broken Pipe error from Varnish health checks In-Reply-To: References: Message-ID: Couldn't a HEAD request solve this? Then nginx wouldn't bother with the body at all, right? This is what we do with our health checks. For example: backend someBackend { .host = "[redacted]"; .port = "80"; .probe = { .interval = 9s; .request = "HEAD /healthcheck HTTP/1.1" "Host: [redacted]" "User-Agent: varnish-health-probe" "Connection: Close" "Accept: */*"; } } ________________________________ From: varnish-misc on behalf of George Sent: Monday, April 17, 2023 10:21 AM To: varnish-misc at varnish-cache.org Subject: Strange Broken Pipe error from Varnish health checks Hi, I have a Varnish/nginx cluster running with varnish-7.1.2-1.el7.x86_64 on CentOS 7. The issue I am having comes from the varnish health checks. I am getting a "broken pipe" error in the nginx error log at random times like below: Apr 10 17:32:46 VARNISH-MASTER nginx_varnish_error: 2023/04/10 17:32:46 [info] 17808#17808: *67626636 writev() failed (32: Broken pipe), client: unix:, server: _, request: "GET /varnish_check HTTP/1.1", host: "0.0.0.0" The strange thing is that this error appears only when Varnish performs the health checks. I have other scripts doing it(nagios, curl, wget, AWS ELB) but those do not show any errors. In addition to this Varnish and nginx where the health checks occur are on the same server and it makes no difference if I use a TCP connection or socket based one. Below are the varnish vcl and nginx locations for the health checks: backend nginx_varnish { .path = "/run/nginx/nginx.sock"; .first_byte_timeout = 600s; .probe = health; } location = /varnish_check { keepalive_timeout 305; return 200 'Varnish Check'; access_log /var/log/nginx/varnish_check.log main; error_log /var/log/nginx/varnish_check_errors.log debug; error_log syslog:server=unix:/run/nginx_log.in.sock,facility=local1,tag=nginx_varnish_error,nohostname info; } Are there any docs I can read about how exactly varnish performs the health checks and what internal processes are involved? Did anyone happen to have similar issues? This is not causing any operational problems for the cluster but it is just something that I want to determine why it is happening because it just should not be happening. Please help THanks in advance. -------------- next part -------------- An HTML attachment was scrubbed... URL: From guillaume.quintard at gmail.com Wed Apr 19 14:22:15 2023 From: guillaume.quintard at gmail.com (Guillaume Quintard) Date: Wed, 19 Apr 2023 07:22:15 -0700 Subject: Possible to disable/inactivate a backend using VCL? In-Reply-To: References: Message-ID: Thank, I think I get it now. How about: backend theBackend none; Here's the relevant documentation: https://varnish-cache.org/docs/trunk/users-guide/vcl-backends.html#the-none-backend It was added in 6.4. Hope that helps. -- Guillaume Quintard On Wed, Apr 19, 2023 at 1:36?AM Batanun B wrote: > Hi Guillaume, > > > I'm curious, if it's completely deactivated what's the benefit of having > it in the vcl? > > It is only intended to be deactivated in production (until we go live). > Our test and staging environments have the backend active. > > > if (false) { > > set req.backend_hint = you_deactivated_backend; > > } > > Thanks, I will test this. > My current prod-specific setup for this backend looks like this: > > backend theBackend { > .host = "localhost"; > .port = "9999"; > .probe = { > .interval = 1h; > } > } > > This seems to be working when testing it locally. It also solves the > problem of having to assign some arbitrary ip or hostname (the actual > backend host for this service hasn't been created in production yet, since > we are several months away from go live), which actually was our main > problem. What do you think about this approach instead? Preferably this > would be a built in feature in Varnish, with a setting "disabled = true" or > similar in the backend definition, and then it would not require any host > or ip to be configured. > > _______________________________________________ > varnish-misc mailing list > varnish-misc at varnish-cache.org > https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc > -------------- next part -------------- An HTML attachment was scrubbed... URL: From guillaume.quintard at gmail.com Wed Apr 19 14:42:23 2023 From: guillaume.quintard at gmail.com (Guillaume Quintard) Date: Wed, 19 Apr 2023 07:42:23 -0700 Subject: Varnish won't start because backend host resolves to too many addresses, but they are all identical IPs In-Reply-To: References: Message-ID: The fact the IPs are identical is weird, but I wouldn't be surprised if the dns entry actually contained 3 identical IPs. > Shouldn't Varnish be able to figure out that in that case it can just choose any one and it will work as expected? Shouldn't your DNS entries be clean? ;-) Honestly, if the IP(s) behind the service name is liable to change, you shouldn't use a dynamic backend because Varnish resolves the IP when the VCL is loaded, so if the IP changes behind your back, Varnish won't follow it, and you'll be screwed. Instead, you should use dynamic backends, of which there are a handful: - dynamic , by UPLEX: it's been around for ages, it's battle-tested, and it's included in the oficial Varnish Docker image - udo+activedns , by Varnish Software: the design is slightly different and allows you to specify pretty much any load-balancing policy you might need. You'll need a subscription but you'll get excellent support (disclaimer, I'm an ex employee) - reqwest , by yours truly: the interface focuses on providing a simple experience and a few bells and whistles (HTTPS, HTTP2, brotli, following redirects) As you can see, the static backend's reluctance to fully handle DNS has been a fertile ground for vmods :-) -- Guillaume Quintard On Wed, Apr 19, 2023 at 1:49?AM Batanun B wrote: > All of the sudden Varnish fails to start in my development environment, > and gives me the following error message: > > Message from VCC-compiler: > Backend host "redacted-hostname": resolves to too many addresses. > Only one IPv4 and one IPv6 are allowed. > Please specify which exact address you want to use, we found all of these: > 555.123.123.3:80 > 555.123.123.3:80 > 555.123.123.3:80 > > I have changed the hostname and the IP above to not expose our server, but > all three IP numbers are 100% identical. Shouldn't Varnish be able to > figure out that in that case it can just choose any one and it will work as > expected? It really should remove duplicates, and only if there are more > than one non-duplicate IP then it should fail. > > The problem is that the backend host is a so called "app service" in > Microsoft Azure, which is basically a platform as a service (PaaS), where > Microsoft handles the networking including the domain name (no user access > it directly). I have no idea why it suddenly resolves to multiple duplicate > IPs. > _______________________________________________ > varnish-misc mailing list > varnish-misc at varnish-cache.org > https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc > -------------- next part -------------- An HTML attachment was scrubbed... URL: From dridi at varni.sh Wed Apr 19 15:38:29 2023 From: dridi at varni.sh (Dridi Boukelmoune) Date: Wed, 19 Apr 2023 15:38:29 +0000 Subject: Varnish won't start because backend host resolves to too many addresses, but they are all identical IPs In-Reply-To: References: Message-ID: On Wed, Apr 19, 2023 at 2:44?PM Guillaume Quintard wrote: > > The fact the IPs are identical is weird, but I wouldn't be surprised if the dns entry actually contained 3 identical IPs. > > > Shouldn't Varnish be able to figure out that in that case it can just choose any one and it will work as expected? > > Shouldn't your DNS entries be clean? ;-) It should, but Varnish could probably make an effort here. From batanun at hotmail.com Wed Apr 19 15:43:07 2023 From: batanun at hotmail.com (Batanun B) Date: Wed, 19 Apr 2023 15:43:07 +0000 Subject: Possible to disable/inactivate a backend using VCL? In-Reply-To: References: Message-ID: > backend theBackend none; > Here's the relevant documentation: https://varnish-cache.org/docs/trunk/users-guide/vcl-backends.html#the-none-backend > It was added in 6.4. Look like exactly what we need! Sadly we are "stuck" on 6.0 until the next LTS version comes. So I think that until then I will use our poor man version of the "none" backend, ie pointing to localhost with an port number that won't give a response. -------------- next part -------------- An HTML attachment was scrubbed... URL: From batanun at hotmail.com Wed Apr 19 16:02:08 2023 From: batanun at hotmail.com (Batanun B) Date: Wed, 19 Apr 2023 16:02:08 +0000 Subject: Varnish won't start because backend host resolves to too many addresses, but they are all identical IPs In-Reply-To: References: Message-ID: > Shouldn't your DNS entries be clean? ;-) Preferably, but I blame Microsoft here ?? The problem went away by itself when I tried starting again like half an hour later or so, so I guess it was a temporary glitch in the matrix. As far as I understand it, the IPs of these machines only change if they are deleted and created again. We do it occasionally in test/staging, and there we can live with Varnish needing to be restarted. In production we don't really delete them once they are properly setup, unless there is some major problem and then a restart of the load balanced varnish servers should not be a concern. Thanks for your vmod suggestions! I will check them out. The dynamic one seems like the only one that supports community edition LTS 6.0. The documentation seems a bit lacking (no full VCL example), but I guess I could use their test cases as examples. ________________________________ From: Guillaume Quintard Sent: Wednesday, April 19, 2023 4:42 PM To: Batanun B Cc: varnish-misc at varnish-cache.org Subject: Re: Varnish won't start because backend host resolves to too many addresses, but they are all identical IPs The fact the IPs are identical is weird, but I wouldn't be surprised if the dns entry actually contained 3 identical IPs. > Shouldn't Varnish be able to figure out that in that case it can just choose any one and it will work as expected? Shouldn't your DNS entries be clean? ;-) Honestly, if the IP(s) behind the service name is liable to change, you shouldn't use a dynamic backend because Varnish resolves the IP when the VCL is loaded, so if the IP changes behind your back, Varnish won't follow it, and you'll be screwed. Instead, you should use dynamic backends, of which there are a handful: - dynamic, by UPLEX: it's been around for ages, it's battle-tested, and it's included in the oficial Varnish Docker image - udo+activedns, by Varnish Software: the design is slightly different and allows you to specify pretty much any load-balancing policy you might need. You'll need a subscription but you'll get excellent support (disclaimer, I'm an ex employee) - reqwest, by yours truly: the interface focuses on providing a simple experience and a few bells and whistles (HTTPS, HTTP2, brotli, following redirects) As you can see, the static backend's reluctance to fully handle DNS has been a fertile ground for vmods :-) -- Guillaume Quintard On Wed, Apr 19, 2023 at 1:49?AM Batanun B > wrote: All of the sudden Varnish fails to start in my development environment, and gives me the following error message: Message from VCC-compiler: Backend host "redacted-hostname": resolves to too many addresses. Only one IPv4 and one IPv6 are allowed. Please specify which exact address you want to use, we found all of these: 555.123.123.3:80 555.123.123.3:80 555.123.123.3:80 I have changed the hostname and the IP above to not expose our server, but all three IP numbers are 100% identical. Shouldn't Varnish be able to figure out that in that case it can just choose any one and it will work as expected? It really should remove duplicates, and only if there are more than one non-duplicate IP then it should fail. The problem is that the backend host is a so called "app service" in Microsoft Azure, which is basically a platform as a service (PaaS), where Microsoft handles the networking including the domain name (no user access it directly). I have no idea why it suddenly resolves to multiple duplicate IPs. _______________________________________________ varnish-misc mailing list varnish-misc at varnish-cache.org https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc -------------- next part -------------- An HTML attachment was scrubbed... URL: From guillaume.quintard at gmail.com Wed Apr 19 16:20:49 2023 From: guillaume.quintard at gmail.com (Guillaume Quintard) Date: Wed, 19 Apr 2023 09:20:49 -0700 Subject: Varnish won't start because backend host resolves to too many addresses, but they are all identical IPs In-Reply-To: References: Message-ID: > The documentation seems a bit lacking (no full VCL example), but I guess I could use their test cases as examples. https://github.com/nigoroll/libvmod-dynamic/blob/master/src/vmod_dynamic.vcc#L538-L583 maybe? I'm sure Nils will pipe up here if you need help, and if you want more synchronous assistance, there's always the discord channel . > The dynamic one seems like the only one that supports community edition LTS 6.0. Yes, of the three, that's the only one that will support that one (VS is focused on the Enterprise version, and I lack the time to port vmods on 6.0 (but I'll welcome the help)). Cheers, -- Guillaume Quintard On Wed, Apr 19, 2023 at 9:02?AM Batanun B wrote: > > Shouldn't your DNS entries be clean? ;-) > > Preferably, but I blame Microsoft here ?? > > The problem went away by itself when I tried starting again like half an > hour later or so, so I guess it was a temporary glitch in the matrix. > > As far as I understand it, the IPs of these machines only change if they > are deleted and created again. We do it occasionally in test/staging, and > there we can live with Varnish needing to be restarted. In production we > don't really delete them once they are properly setup, unless there is some > major problem and then a restart of the load balanced varnish servers > should not be a concern. > > Thanks for your vmod suggestions! I will check them out. The dynamic one > seems like the only one that supports community edition LTS 6.0. The > documentation seems a bit lacking (no full VCL example), but I guess I > could use their test cases as examples. > > ------------------------------ > *From:* Guillaume Quintard > *Sent:* Wednesday, April 19, 2023 4:42 PM > *To:* Batanun B > *Cc:* varnish-misc at varnish-cache.org > *Subject:* Re: Varnish won't start because backend host resolves to too > many addresses, but they are all identical IPs > > The fact the IPs are identical is weird, but I wouldn't be surprised if > the dns entry actually contained 3 identical IPs. > > > Shouldn't Varnish be able to figure out that in that case it can just > choose any one and it will work as expected? > > Shouldn't your DNS entries be clean? ;-) > > Honestly, if the IP(s) behind the service name is liable to change, you > shouldn't use a dynamic backend because Varnish resolves the IP when the > VCL is loaded, so if the IP changes behind your back, Varnish won't follow > it, and you'll be screwed. > Instead, you should use dynamic backends, of which there are a handful: > - dynamic , by UPLEX: it's > been around for ages, it's battle-tested, and it's included in the oficial > Varnish Docker image > - udo+activedns > , > by Varnish Software: the design is slightly different and allows you to > specify pretty much any load-balancing policy you might need. You'll need a > subscription but you'll get excellent support (disclaimer, I'm an ex > employee) > - reqwest > , > by yours truly: the interface focuses on providing a simple experience and > a few bells and whistles (HTTPS, HTTP2, brotli, following redirects) > > As you can see, the static backend's reluctance to fully handle DNS has > been a fertile ground for vmods :-) > > -- > Guillaume Quintard > > > On Wed, Apr 19, 2023 at 1:49?AM Batanun B wrote: > > All of the sudden Varnish fails to start in my development environment, > and gives me the following error message: > > Message from VCC-compiler: > Backend host "redacted-hostname": resolves to too many addresses. > Only one IPv4 and one IPv6 are allowed. > Please specify which exact address you want to use, we found all of these: > 555.123.123.3:80 > 555.123.123.3:80 > 555.123.123.3:80 > > I have changed the hostname and the IP above to not expose our server, but > all three IP numbers are 100% identical. Shouldn't Varnish be able to > figure out that in that case it can just choose any one and it will work as > expected? It really should remove duplicates, and only if there are more > than one non-duplicate IP then it should fail. > > The problem is that the backend host is a so called "app service" in > Microsoft Azure, which is basically a platform as a service (PaaS), where > Microsoft handles the networking including the domain name (no user access > it directly). I have no idea why it suddenly resolves to multiple duplicate > IPs. > _______________________________________________ > varnish-misc mailing list > varnish-misc at varnish-cache.org > https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From batanun at hotmail.com Wed Apr 19 16:22:49 2023 From: batanun at hotmail.com (Batanun B) Date: Wed, 19 Apr 2023 16:22:49 +0000 Subject: Is there any "try catch" functionality in VCL? If not, how to handle runtime errors in vcl_init? Message-ID: Hi, We use the vmod crypto to verify cryptographic signatures for some of our traffic. When testing, the public key was hard coded in the VCL, but before we start using this feature in production we will switch to reading the public key from a file on disk. This file is generated on server startup, by fetching it from an Azure keyvault. Now, the problem I'm picturing here is that this fetching of the public key can fail, or the key can be corrupt or empty, maybe by user error. Or the key could be valid, but the format of the key happens to be unsupported by the vmod crypto. So, even if we do our best to validate the key, in theory it could pass all our tests but still fail when we give it to the vmod crypto. And if that happens, Varnish won't start because the vmod crypto is initiated with the public key in vcl_init, like this: sub vcl_init { new cryptoVerifier = crypto.verifier(sha256, std.fileread("/path/to/public.key")); } What I would prefer to happen if the key is rejected, is that vcl_init goes through without failure, and then the requests that use the cryptoVerifier will fail, but all other traffic (like 99%) still works. Can we achieve this somehow? Like some try-catch functionallity? If not, is there some other way to handle this that doesn't cause Varnish to die on startup? -------------- next part -------------- An HTML attachment was scrubbed... URL: From dridi at varni.sh Wed Apr 19 16:25:38 2023 From: dridi at varni.sh (Dridi Boukelmoune) Date: Wed, 19 Apr 2023 16:25:38 +0000 Subject: Possible to disable/inactivate a backend using VCL? In-Reply-To: References: Message-ID: On Wed, Apr 19, 2023 at 3:45?PM Batanun B wrote: > > > backend theBackend none; > > Here's the relevant documentation: https://varnish-cache.org/docs/trunk/users-guide/vcl-backends.html#the-none-backend > > It was added in 6.4. > > Look like exactly what we need! Sadly we are "stuck" on 6.0 until the next LTS version comes. So I think that until then I will use our poor man version of the "none" backend, ie pointing to localhost with an port number that won't give a response. It was back-ported to 6.0, which is not an LTS branch limited to bug fixes ;) https://varnish-cache.org/docs/6.0/users-guide/vcl-backends.html Dridi From dridi at varni.sh Wed Apr 19 16:29:08 2023 From: dridi at varni.sh (Dridi Boukelmoune) Date: Wed, 19 Apr 2023 16:29:08 +0000 Subject: Is there any "try catch" functionality in VCL? If not, how to handle runtime errors in vcl_init? In-Reply-To: References: Message-ID: On Wed, Apr 19, 2023 at 4:25?PM Batanun B wrote: > > Hi, > > We use the vmod crypto to verify cryptographic signatures for some of our traffic. When testing, the public key was hard coded in the VCL, but before we start using this feature in production we will switch to reading the public key from a file on disk. This file is generated on server startup, by fetching it from an Azure keyvault. > > Now, the problem I'm picturing here is that this fetching of the public key can fail, or the key can be corrupt or empty, maybe by user error. Or the key could be valid, but the format of the key happens to be unsupported by the vmod crypto. So, even if we do our best to validate the key, in theory it could pass all our tests but still fail when we give it to the vmod crypto. And if that happens, Varnish won't start because the vmod crypto is initiated with the public key in vcl_init, like this: > > sub vcl_init { > new cryptoVerifier = crypto.verifier(sha256, std.fileread("/path/to/public.key")); > } > > What I would prefer to happen if the key is rejected, is that vcl_init goes through without failure, and then the requests that use the cryptoVerifier will fail, but all other traffic (like 99%) still works. Can we achieve this somehow? Like some try-catch functionallity? If not, is there some other way to handle this that doesn't cause Varnish to die on startup? It's the VMOD author you should ask to have an option to ignore public key errors. This is a constructor, and even if we had a try-catch kind of construct in the language, I don't think we would make this one recoverable. Dridi From batanun at hotmail.com Wed Apr 19 16:36:36 2023 From: batanun at hotmail.com (Batanun B) Date: Wed, 19 Apr 2023 16:36:36 +0000 Subject: Possible to disable/inactivate a backend using VCL? In-Reply-To: References: Message-ID: > It was back-ported to 6.0, which is not an LTS branch limited to bug fixes ;) > > https://varnish-cache.org/docs/6.0/users-guide/vcl-backends.html Thanks! Wow, I can't believe that I could miss that. I thought I read that specific page, as well as searched on Google, but I guess I was too focused on looking for terms like "disabled" or "inactivated". -------------- next part -------------- An HTML attachment was scrubbed... URL: From batanun at hotmail.com Wed Apr 19 16:45:01 2023 From: batanun at hotmail.com (Batanun B) Date: Wed, 19 Apr 2023 16:45:01 +0000 Subject: Is there any "try catch" functionality in VCL? If not, how to handle runtime errors in vcl_init? In-Reply-To: References: Message-ID: > It's the VMOD author you should ask to have an option to ignore public > key errors. Well, I'm usually of the mindset that if a problem can be handled in a generic way by the language/platform/framework, then one should avoid requiring each and every custom vmod/plugin/library to handle it individually. And I'm also of the mindset that pretty much any non-trivial code can fail, and the calling code should be able to catch that if needed. :) > This is a constructor, and even if we had a try-catch kind of > construct in the language, I don't think we would make this one > recoverable. In my mind, with a try-catch I could handle it like this: try { new cryptoVerifier = crypto.verifier(sha256, std.fileread("/path/to/public.key")); } catch (error) { // log error... // then try with with a hard coded known safe key, but that will fail when checking the signature new cryptoVerifier = crypto.verifier(sha256, (sha256, {" -----BEGIN PUBLIC KEY----- ... -----END PUBLIC KEY----- "}); } With this approach, Varnish will start like normal. And the only requests failing will be the ones using the cryptoVerifier. -------------- next part -------------- An HTML attachment was scrubbed... URL: From batanun at hotmail.com Wed Apr 19 16:48:32 2023 From: batanun at hotmail.com (Batanun B) Date: Wed, 19 Apr 2023 16:48:32 +0000 Subject: Varnish won't start because backend host resolves to too many addresses, but they are all identical IPs In-Reply-To: References: Message-ID: > https://github.com/nigoroll/libvmod-dynamic/blob/master/src/vmod_dynamic.vcc#L538-L583 > maybe? > > I'm sure Nils will pipe up here if you need help, and if you want more > synchronous assistance, there's always the discord channel > . Thanks! :) -------------- next part -------------- An HTML attachment was scrubbed... URL: From batanun at hotmail.com Wed Apr 19 17:04:18 2023 From: batanun at hotmail.com (Batanun B) Date: Wed, 19 Apr 2023 17:04:18 +0000 Subject: Is there any "try catch" functionality in VCL? If not, how to handle runtime errors in vcl_init? In-Reply-To: References: Message-ID: Just to explain my concern a bit. The worst case?scenario in production, that I very much would like to avoid, could look something like this: 1. Something happens with our public key, so that Varnish won't be able to start after getting the new faulty key. Already running servers will continue to run, but no new servers can be initiated. 2. Before we have been able to fix the problem with the public key, some other problem happens in Varnish, and all running Varnish servers dies. 3. We end up with no working Varnish servers running, and not being able to start any new ones. 4. All our websites are down.