From batanun at hotmail.com  Fri Apr 14 11:51:06 2023
From: batanun at hotmail.com (Batanun B)
Date: Fri, 14 Apr 2023 11:51:06 +0000
Subject: Possible to disable/inactivate a backend using VCL?
Message-ID: <PR3P190MB0922E70F10984632A678E67AAC999@PR3P190MB0922.EURP190.PROD.OUTLOOK.COM>

Hi,

We are currently working on a new feature that won't go live for several months still. This new feature has it's own backend in Varnish. Most of our VCL code is identical for all environments, and this code refers to the new backend, so it needs to be defined otherwise Varnish won't start. But in production we don't want to show anything of this feature. And we would like to have this backend completely disabled or inactivated in Varnish. Can we do that using VCL? Like forcing the health to be sick, or something similar. We would prefer to keep this inside the backend declaration if possible, and we would also prefer somethink not too "hackish" (like pointing it to a dead IP).

Does Varnish has any recommended approach for this? I know there is a cli command to set the health, but as I said we would really prefer doing this using VCL only.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/varnish-misc/attachments/20230414/0a1536a2/attachment-0001.html>

From guillaume.quintard at gmail.com  Fri Apr 14 15:52:41 2023
From: guillaume.quintard at gmail.com (Guillaume Quintard)
Date: Fri, 14 Apr 2023 08:52:41 -0700
Subject: Possible to disable/inactivate a backend using VCL?
In-Reply-To: <PR3P190MB0922E70F10984632A678E67AAC999@PR3P190MB0922.EURP190.PROD.OUTLOOK.COM>
References: <PR3P190MB0922E70F10984632A678E67AAC999@PR3P190MB0922.EURP190.PROD.OUTLOOK.COM>
Message-ID: <CAOrPorkq4h5qRigFphQ5i5ckgoyx=_LvS1CWob2kPzHFdar1LQ@mail.gmail.com>

Hi Batanun,

I'm curious, if it's completely deactivated what's the benefit of having it
in the vcl?

To answer your question: cheat! Add this somewhere in your vcl:

sub vcl_recv {
    # the if statement will never work, but the backend is referenced, so
the compiler won't bother you
    if (false) {
        set req.backend_hint = you_deactivated_backend;
    }
}

You could also start varnish with `-p vcc_feature=-err_unref` but I don't
recommend it as it'll affect all the unreferenced symbols, even the ones
you may care about.

Hope that helps

-- 
Guillaume Quintard


On Fri, Apr 14, 2023 at 4:53?AM Batanun B <batanun at hotmail.com> wrote:

> Hi,
>
> We are currently working on a new feature that won't go live for several
> months still. This new feature has it's own backend in Varnish. Most of our
> VCL code is identical for all environments, and this code refers to the new
> backend, so it needs to be defined otherwise Varnish won't start. But in
> production we don't want to show anything of this feature. And we would
> like to have this backend completely disabled or inactivated in Varnish.
> Can we do that using VCL? Like forcing the health to be sick, or something
> similar. We would prefer to keep this inside the backend declaration if
> possible, and we would also prefer somethink not too "hackish" (like
> pointing it to a dead IP).
>
> Does Varnish has any recommended approach for this? I know there is a cli
> command to set the health, but as I said we would really prefer doing this
> using VCL only.
> _______________________________________________
> varnish-misc mailing list
> varnish-misc at varnish-cache.org
> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/varnish-misc/attachments/20230414/b71870e9/attachment.html>

From izghitu at gmail.com  Mon Apr 17 08:21:17 2023
From: izghitu at gmail.com (George)
Date: Mon, 17 Apr 2023 11:21:17 +0300
Subject: Strange Broken Pipe error from Varnish health checks
Message-ID: <CABd=SGWQaTntnh1prbnYWUF5L1QLK5ko++Nee3ApOsVe=oE05A@mail.gmail.com>

Hi,

I have a Varnish/nginx cluster running with varnish-7.1.2-1.el7.x86_64 on
CentOS 7.

The issue I am having comes from the varnish health checks. I am getting a
"broken pipe" error in the nginx error log at random times like below:
Apr 10 17:32:46 VARNISH-MASTER nginx_varnish_error: 2023/04/10 17:32:46
[info] 17808#17808: *67626636 writev() failed (32: Broken pipe), client:
unix:, server: _, request: "GET /varnish_check HTTP/1.1", host: "0.0.0.0"

The strange thing is that this error appears only when Varnish performs the
health checks. I have other scripts doing it(nagios, curl, wget, AWS ELB)
but those do not show any errors. In addition to this Varnish and nginx
where the health checks occur are on the same server and it makes no
difference if I use a TCP connection or socket based one.

Below are the varnish vcl and nginx locations for the health checks:
backend nginx_varnish {
                   .path = "/run/nginx/nginx.sock";
                   .first_byte_timeout = 600s;
                   .probe = health;
        }

location = /varnish_check {
                keepalive_timeout 305;
                return 200 'Varnish Check';
                access_log /var/log/nginx/varnish_check.log main;
                error_log /var/log/nginx/varnish_check_errors.log debug;
                error_log
syslog:server=unix:/run/nginx_log.in.sock,facility=local1,tag=nginx_varnish_error,nohostname
info;
        }

Are there any docs I can read about how exactly varnish performs the health
checks and what internal processes are involved?
Did anyone happen to have similar issues? This is not causing any
operational problems for the cluster but it is just something that I want
to determine why it is happening because it just should not be happening.

Please help
THanks in advance.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/varnish-misc/attachments/20230417/6fe44cfa/attachment.html>

From guillaume.quintard at gmail.com  Mon Apr 17 14:31:48 2023
From: guillaume.quintard at gmail.com (Guillaume Quintard)
Date: Mon, 17 Apr 2023 07:31:48 -0700
Subject: Strange Broken Pipe error from Varnish health checks
In-Reply-To: <CABd=SGWQaTntnh1prbnYWUF5L1QLK5ko++Nee3ApOsVe=oE05A@mail.gmail.com>
References: <CABd=SGWQaTntnh1prbnYWUF5L1QLK5ko++Nee3ApOsVe=oE05A@mail.gmail.com>
Message-ID: <CAOrPorn9fn1dHP98CWUfiMMDMSyGr+64vHyu9QXgfU5SA4LxKA@mail.gmail.com>

Hi George,

Just to be sure, how is the probe "health" defined in your VCL?

Cheers,

-- 
Guillaume Quintard


On Mon, Apr 17, 2023 at 1:23?AM George <izghitu at gmail.com> wrote:

> Hi,
>
> I have a Varnish/nginx cluster running with varnish-7.1.2-1.el7.x86_64 on
> CentOS 7.
>
> The issue I am having comes from the varnish health checks. I am getting a
> "broken pipe" error in the nginx error log at random times like below:
> Apr 10 17:32:46 VARNISH-MASTER nginx_varnish_error: 2023/04/10 17:32:46
> [info] 17808#17808: *67626636 writev() failed (32: Broken pipe), client:
> unix:, server: _, request: "GET /varnish_check HTTP/1.1", host: "0.0.0.0"
>
> The strange thing is that this error appears only when Varnish performs
> the health checks. I have other scripts doing it(nagios, curl, wget, AWS
> ELB) but those do not show any errors. In addition to this Varnish and
> nginx where the health checks occur are on the same server and it makes no
> difference if I use a TCP connection or socket based one.
>
> Below are the varnish vcl and nginx locations for the health checks:
> backend nginx_varnish {
>                    .path = "/run/nginx/nginx.sock";
>                    .first_byte_timeout = 600s;
>                    .probe = health;
>         }
>
> location = /varnish_check {
>                 keepalive_timeout 305;
>                 return 200 'Varnish Check';
>                 access_log /var/log/nginx/varnish_check.log main;
>                 error_log /var/log/nginx/varnish_check_errors.log debug;
>                 error_log
> syslog:server=unix:/run/nginx_log.in.sock,facility=local1,tag=nginx_varnish_error,nohostname
> info;
>         }
>
> Are there any docs I can read about how exactly varnish performs the
> health checks and what internal processes are involved?
> Did anyone happen to have similar issues? This is not causing any
> operational problems for the cluster but it is just something that I want
> to determine why it is happening because it just should not be happening.
>
> Please help
> THanks in advance.
>
> _______________________________________________
> varnish-misc mailing list
> varnish-misc at varnish-cache.org
> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/varnish-misc/attachments/20230417/ad34c346/attachment.html>

From izghitu at gmail.com  Mon Apr 17 14:47:20 2023
From: izghitu at gmail.com (George)
Date: Mon, 17 Apr 2023 17:47:20 +0300
Subject: Strange Broken Pipe error from Varnish health checks
In-Reply-To: <CAOrPorn9fn1dHP98CWUfiMMDMSyGr+64vHyu9QXgfU5SA4LxKA@mail.gmail.com>
References: <CABd=SGWQaTntnh1prbnYWUF5L1QLK5ko++Nee3ApOsVe=oE05A@mail.gmail.com>
 <CAOrPorn9fn1dHP98CWUfiMMDMSyGr+64vHyu9QXgfU5SA4LxKA@mail.gmail.com>
Message-ID: <CABd=SGXP+hZ481YmJ5q07Z1VTc3UwTHuH7hKMoN+BPq28KxRzg@mail.gmail.com>

Hi,

Below is the probe "health", I forgot to send it the first time:
probe health {
                   .url = "/varnish_check";
                   .timeout = 2s;
                .interval = 5s;
                .window = 3;
                .threshold = 2;
      }

Thanks

?n lun., 17 apr. 2023 la 17:32, Guillaume Quintard <
guillaume.quintard at gmail.com> a scris:

> Hi George,
>
> Just to be sure, how is the probe "health" defined in your VCL?
>
> Cheers,
>
> --
> Guillaume Quintard
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/varnish-misc/attachments/20230417/1b32e6b7/attachment-0001.html>

From guillaume.quintard at gmail.com  Mon Apr 17 15:46:56 2023
From: guillaume.quintard at gmail.com (Guillaume Quintard)
Date: Mon, 17 Apr 2023 08:46:56 -0700
Subject: Strange Broken Pipe error from Varnish health checks
In-Reply-To: <CABd=SGXP+hZ481YmJ5q07Z1VTc3UwTHuH7hKMoN+BPq28KxRzg@mail.gmail.com>
References: <CABd=SGWQaTntnh1prbnYWUF5L1QLK5ko++Nee3ApOsVe=oE05A@mail.gmail.com>
 <CAOrPorn9fn1dHP98CWUfiMMDMSyGr+64vHyu9QXgfU5SA4LxKA@mail.gmail.com>
 <CABd=SGXP+hZ481YmJ5q07Z1VTc3UwTHuH7hKMoN+BPq28KxRzg@mail.gmail.com>
Message-ID: <CAOrPormEWzzPUxJqMMuHtD29A13tUbahLdPZ1P4UJidNbQg3+g@mail.gmail.com>

Thanks, I looked at the code quickly and I'd venture that maybe the
/varnish_check is a bit too large and doesn't fit Varnish's probe buffer
(we only care about the status line of the response anyway), so Varnish
closes the connection while nginx isn't done yet.

If it's that, it's not very polite, but it's harmless.
-- 
Guillaume Quintard


On Mon, Apr 17, 2023 at 7:49?AM George <izghitu at gmail.com> wrote:

> Hi,
>
> Below is the probe "health", I forgot to send it the first time:
> probe health {
>                    .url = "/varnish_check";
>                    .timeout = 2s;
>                 .interval = 5s;
>                 .window = 3;
>                 .threshold = 2;
>       }
>
> Thanks
>
> ?n lun., 17 apr. 2023 la 17:32, Guillaume Quintard <
> guillaume.quintard at gmail.com> a scris:
>
>> Hi George,
>>
>> Just to be sure, how is the probe "health" defined in your VCL?
>>
>> Cheers,
>>
>> --
>> Guillaume Quintard
>>
>> _______________________________________________
> varnish-misc mailing list
> varnish-misc at varnish-cache.org
> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/varnish-misc/attachments/20230417/68e1367a/attachment.html>

From izghitu at gmail.com  Mon Apr 17 16:10:56 2023
From: izghitu at gmail.com (George)
Date: Mon, 17 Apr 2023 19:10:56 +0300
Subject: Strange Broken Pipe error from Varnish health checks
In-Reply-To: <CAOrPormEWzzPUxJqMMuHtD29A13tUbahLdPZ1P4UJidNbQg3+g@mail.gmail.com>
References: <CABd=SGWQaTntnh1prbnYWUF5L1QLK5ko++Nee3ApOsVe=oE05A@mail.gmail.com>
 <CAOrPorn9fn1dHP98CWUfiMMDMSyGr+64vHyu9QXgfU5SA4LxKA@mail.gmail.com>
 <CABd=SGXP+hZ481YmJ5q07Z1VTc3UwTHuH7hKMoN+BPq28KxRzg@mail.gmail.com>
 <CAOrPormEWzzPUxJqMMuHtD29A13tUbahLdPZ1P4UJidNbQg3+g@mail.gmail.com>
Message-ID: <CABd=SGWDSKkRqAVAWOZBNR+knnaJND9=F0JLx31f=qeBQgf5AA@mail.gmail.com>

Hi,

In our case the response body is few bytes (13), header+body is 170 bytes.
Can this be a bug related to something else perhaps?

Please let me know.
Thanks

?n lun., 17 apr. 2023 la 18:47, Guillaume Quintard <
guillaume.quintard at gmail.com> a scris:

> Thanks, I looked at the code quickly and I'd venture that maybe the
> /varnish_check is a bit too large and doesn't fit Varnish's probe buffer
> (we only care about the status line of the response anyway), so Varnish
> closes the connection while nginx isn't done yet.
>
> If it's that, it's not very polite, but it's harmless.
> --
> Guillaume Quintard
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/varnish-misc/attachments/20230417/f900c928/attachment.html>

From guillaume.quintard at gmail.com  Mon Apr 17 16:15:00 2023
From: guillaume.quintard at gmail.com (Guillaume Quintard)
Date: Mon, 17 Apr 2023 09:15:00 -0700
Subject: Strange Broken Pipe error from Varnish health checks
In-Reply-To: <CABd=SGWDSKkRqAVAWOZBNR+knnaJND9=F0JLx31f=qeBQgf5AA@mail.gmail.com>
References: <CABd=SGWQaTntnh1prbnYWUF5L1QLK5ko++Nee3ApOsVe=oE05A@mail.gmail.com>
 <CAOrPorn9fn1dHP98CWUfiMMDMSyGr+64vHyu9QXgfU5SA4LxKA@mail.gmail.com>
 <CABd=SGXP+hZ481YmJ5q07Z1VTc3UwTHuH7hKMoN+BPq28KxRzg@mail.gmail.com>
 <CAOrPormEWzzPUxJqMMuHtD29A13tUbahLdPZ1P4UJidNbQg3+g@mail.gmail.com>
 <CABd=SGWDSKkRqAVAWOZBNR+knnaJND9=F0JLx31f=qeBQgf5AA@mail.gmail.com>
Message-ID: <CAOrPorn_8_Hj3igde=vthY1DisTrz09BCJvrfXEwiUAaOZ1Hkw@mail.gmail.com>

That code hasn't moved in a while, so I'd be surprised to see a bug there,
but that's always possible.
Any chance you could get a tcpdump of a probe request (from connection to
disconnection) so we can see what's going on?
-- 
Guillaume Quintard


On Mon, Apr 17, 2023 at 9:12?AM George <izghitu at gmail.com> wrote:

> Hi,
>
> In our case the response body is few bytes (13), header+body is 170 bytes.
> Can this be a bug related to something else perhaps?
>
> Please let me know.
> Thanks
>
> ?n lun., 17 apr. 2023 la 18:47, Guillaume Quintard <
> guillaume.quintard at gmail.com> a scris:
>
>> Thanks, I looked at the code quickly and I'd venture that maybe the
>> /varnish_check is a bit too large and doesn't fit Varnish's probe buffer
>> (we only care about the status line of the response anyway), so Varnish
>> closes the connection while nginx isn't done yet.
>>
>> If it's that, it's not very polite, but it's harmless.
>> --
>> Guillaume Quintard
>>
>> _______________________________________________
> varnish-misc mailing list
> varnish-misc at varnish-cache.org
> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/varnish-misc/attachments/20230417/d558b919/attachment.html>

From izghitu at gmail.com  Mon Apr 17 19:50:16 2023
From: izghitu at gmail.com (George)
Date: Mon, 17 Apr 2023 22:50:16 +0300
Subject: Strange Broken Pipe error from Varnish health checks
In-Reply-To: <CAOrPorn_8_Hj3igde=vthY1DisTrz09BCJvrfXEwiUAaOZ1Hkw@mail.gmail.com>
References: <CABd=SGWQaTntnh1prbnYWUF5L1QLK5ko++Nee3ApOsVe=oE05A@mail.gmail.com>
 <CAOrPorn9fn1dHP98CWUfiMMDMSyGr+64vHyu9QXgfU5SA4LxKA@mail.gmail.com>
 <CABd=SGXP+hZ481YmJ5q07Z1VTc3UwTHuH7hKMoN+BPq28KxRzg@mail.gmail.com>
 <CAOrPormEWzzPUxJqMMuHtD29A13tUbahLdPZ1P4UJidNbQg3+g@mail.gmail.com>
 <CABd=SGWDSKkRqAVAWOZBNR+knnaJND9=F0JLx31f=qeBQgf5AA@mail.gmail.com>
 <CAOrPorn_8_Hj3igde=vthY1DisTrz09BCJvrfXEwiUAaOZ1Hkw@mail.gmail.com>
Message-ID: <CABd=SGVUoGyyYrzJL5ruhVe4gpRW8gVz198cBTgQaOtuNzMvxQ@mail.gmail.com>

I am not sure how a tcpdump will help in any way. Since all the health
check requests are virtually the same. Since the varnish thread that probes
the backend dies every once in a while then it will never show anything
with tcpdump or I virtually have no way to know when it will die in order
to capture an incomplete request using tcpdump.

Please advise.


?n lun., 17 apr. 2023 la 19:15, Guillaume Quintard <
guillaume.quintard at gmail.com> a scris:

> That code hasn't moved in a while, so I'd be surprised to see a bug there,
> but that's always possible.
> Any chance you could get a tcpdump of a probe request (from connection to
> disconnection) so we can see what's going on?
> --
> Guillaume Quintard
>
>
> On Mon, Apr 17, 2023 at 9:12?AM George <izghitu at gmail.com> wrote:
>
>> Hi,
>>
>> In our case the response body is few bytes (13), header+body is 170
>> bytes. Can this be a bug related to something else perhaps?
>>
>> Please let me know.
>> Thanks
>>
>> ?n lun., 17 apr. 2023 la 18:47, Guillaume Quintard <
>> guillaume.quintard at gmail.com> a scris:
>>
>>> Thanks, I looked at the code quickly and I'd venture that maybe the
>>> /varnish_check is a bit too large and doesn't fit Varnish's probe buffer
>>> (we only care about the status line of the response anyway), so Varnish
>>> closes the connection while nginx isn't done yet.
>>>
>>> If it's that, it's not very polite, but it's harmless.
>>> --
>>> Guillaume Quintard
>>>
>>> _______________________________________________
>> varnish-misc mailing list
>> varnish-misc at varnish-cache.org
>> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/varnish-misc/attachments/20230417/be9bd24f/attachment-0001.html>

From dridi at varni.sh  Tue Apr 18 10:23:01 2023
From: dridi at varni.sh (Dridi Boukelmoune)
Date: Tue, 18 Apr 2023 10:23:01 +0000
Subject: Strange Broken Pipe error from Varnish health checks
In-Reply-To: <CAOrPorn_8_Hj3igde=vthY1DisTrz09BCJvrfXEwiUAaOZ1Hkw@mail.gmail.com>
References: <CABd=SGWQaTntnh1prbnYWUF5L1QLK5ko++Nee3ApOsVe=oE05A@mail.gmail.com>
 <CAOrPorn9fn1dHP98CWUfiMMDMSyGr+64vHyu9QXgfU5SA4LxKA@mail.gmail.com>
 <CABd=SGXP+hZ481YmJ5q07Z1VTc3UwTHuH7hKMoN+BPq28KxRzg@mail.gmail.com>
 <CAOrPormEWzzPUxJqMMuHtD29A13tUbahLdPZ1P4UJidNbQg3+g@mail.gmail.com>
 <CABd=SGWDSKkRqAVAWOZBNR+knnaJND9=F0JLx31f=qeBQgf5AA@mail.gmail.com>
 <CAOrPorn_8_Hj3igde=vthY1DisTrz09BCJvrfXEwiUAaOZ1Hkw@mail.gmail.com>
Message-ID: <CABoVN9C2j5jiH_PzB+V5dwFVaXz9h320Nx_3=h-BCimTL7ukaw@mail.gmail.com>

On Mon, Apr 17, 2023 at 4:17?PM Guillaume Quintard
<guillaume.quintard at gmail.com> wrote:
>
> That code hasn't moved in a while, so I'd be surprised to see a bug there, but that's always possible.
> Any chance you could get a tcpdump of a probe request (from connection to disconnection) so we can see what's going on?

It has: https://github.com/varnishcache/varnish-cache/pull/3886

But it shouldn't have changed the default behavior.

Dridi

From walid.boudebouda at varnish-software.com  Tue Apr 18 10:41:45 2023
From: walid.boudebouda at varnish-software.com (walid boudebouda)
Date: Tue, 18 Apr 2023 12:41:45 +0200
Subject: Strange Broken Pipe error from Varnish health checks
In-Reply-To: <CABoVN9C2j5jiH_PzB+V5dwFVaXz9h320Nx_3=h-BCimTL7ukaw@mail.gmail.com>
References: <CABd=SGWQaTntnh1prbnYWUF5L1QLK5ko++Nee3ApOsVe=oE05A@mail.gmail.com>
 <CAOrPorn9fn1dHP98CWUfiMMDMSyGr+64vHyu9QXgfU5SA4LxKA@mail.gmail.com>
 <CABd=SGXP+hZ481YmJ5q07Z1VTc3UwTHuH7hKMoN+BPq28KxRzg@mail.gmail.com>
 <CAOrPormEWzzPUxJqMMuHtD29A13tUbahLdPZ1P4UJidNbQg3+g@mail.gmail.com>
 <CABd=SGWDSKkRqAVAWOZBNR+knnaJND9=F0JLx31f=qeBQgf5AA@mail.gmail.com>
 <CAOrPorn_8_Hj3igde=vthY1DisTrz09BCJvrfXEwiUAaOZ1Hkw@mail.gmail.com>
 <CABoVN9C2j5jiH_PzB+V5dwFVaXz9h320Nx_3=h-BCimTL7ukaw@mail.gmail.com>
Message-ID: <CABs7ENmRBC6ratgLWM1nL2cQ3ESp14Hh8pM3CeNUnawtWkDnag@mail.gmail.com>

>
>
> It has: https://github.com/varnishcache/varnish-cache/pull/3886
>

This was introduced in 7.3 and he is running 7.1

Walid



On Tue, Apr 18, 2023 at 12:24?PM Dridi Boukelmoune <dridi at varni.sh> wrote:

> On Mon, Apr 17, 2023 at 4:17?PM Guillaume Quintard
> <guillaume.quintard at gmail.com> wrote:
> >
> > That code hasn't moved in a while, so I'd be surprised to see a bug
> there, but that's always possible.
> > Any chance you could get a tcpdump of a probe request (from connection
> to disconnection) so we can see what's going on?
>
> It has: https://github.com/varnishcache/varnish-cache/pull/3886
>
> But it shouldn't have changed the default behavior.
>
> Dridi
> _______________________________________________
> varnish-misc mailing list
> varnish-misc at varnish-cache.org
> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/varnish-misc/attachments/20230418/f67d2f41/attachment.html>

From kokoniimasu at gmail.com  Tue Apr 18 15:03:22 2023
From: kokoniimasu at gmail.com (kokoniimasu)
Date: Wed, 19 Apr 2023 00:03:22 +0900
Subject: Improved github issue templates
Message-ID: <CAJihfGAskzih6zRNPsMV5ebVAdQPJ9np14K6msGTFUJNn25y0g@mail.gmail.com>

Hi,

Varnish already have a issue template, but sometimes I see people ignore it
and post questions, etc. (they are closed right away)

I saw a post on Twitter about the great categorization of issues in
ImageMagick.

https://twitter.com/yoya/status/1644160988600733696

I thought this was very nice and made a sample to see if it could be done
in Varnish as well.
I think that by putting in a category, we can guide them to the right place.

https://github.com/xcir/ghsandbox/issues
(Feel free to create issues for testing.)

When a new issue is made, the category will be displayed.
Bug report required each information (Expected Behavior, Current
Behavior...)

Of course, I understand that it is also used as a TODO, so I also allow a
free format description with "Don't see your issue here? Open a blank
issue." at the bottom of the issue creation screen
A warning is also included here to prevent unintended use.

https://github.com/xcir/ghsandbox/issues/new

I am thinking of submitting a PR if this might be useful, what do you think?

config is here.
https://github.com/xcir/ghsandbox/tree/main/.github/ISSUE_TEMPLATE

I'd be happy to be helpful.

Regards,

--
Shohei Tanaka(@xcir)
https://blog.xcir.net/ (JP)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/varnish-misc/attachments/20230419/a7babe8a/attachment.html>

From dridi at varni.sh  Tue Apr 18 16:11:05 2023
From: dridi at varni.sh (Dridi Boukelmoune)
Date: Tue, 18 Apr 2023 16:11:05 +0000
Subject: Improved github issue templates
In-Reply-To: <CAJihfGAskzih6zRNPsMV5ebVAdQPJ9np14K6msGTFUJNn25y0g@mail.gmail.com>
References: <CAJihfGAskzih6zRNPsMV5ebVAdQPJ9np14K6msGTFUJNn25y0g@mail.gmail.com>
Message-ID: <CABoVN9AUDSYFiE3+U4gLMn_1mmoG5KKHP1AyG3JQG6tMNHx+8w@mail.gmail.com>

On Tue, Apr 18, 2023 at 3:05?PM kokoniimasu <kokoniimasu at gmail.com> wrote:
>
> Hi,
>
> Varnish already have a issue template, but sometimes I see people ignore it and post questions, etc. (they are closed right away)
>
> I saw a post on Twitter about the great categorization of issues in ImageMagick.
>
> https://twitter.com/yoya/status/1644160988600733696
>
> I thought this was very nice and made a sample to see if it could be done in Varnish as well.
> I think that by putting in a category, we can guide them to the right place.
>
> https://github.com/xcir/ghsandbox/issues
> (Feel free to create issues for testing.)
>
> When a new issue is made, the category will be displayed.
> Bug report required each information (Expected Behavior, Current Behavior...)
>
> Of course, I understand that it is also used as a TODO, so I also allow a free format description with "Don't see your issue here? Open a blank issue." at the bottom of the issue creation screen

If we can help it, I'd rather not have the blank issue link.

> A warning is also included here to prevent unintended use.
>
> https://github.com/xcir/ghsandbox/issues/new
>
> I am thinking of submitting a PR if this might be useful, what do you think?

YES PLEASE!

I have had this deep in my backlog for a long time now, and never got
around to fishing it out.

I didn't even know that issue templates could include external links.

> config is here.
> https://github.com/xcir/ghsandbox/tree/main/.github/ISSUE_TEMPLATE
>
> I'd be happy to be helpful.

Feel free to refresh the current template. Maybe we don't need as many
markdown comments as we currently have today with the categorization
you bring upfront.

Many thanks!
Dridi

From izghitu at gmail.com  Tue Apr 18 16:41:54 2023
From: izghitu at gmail.com (George)
Date: Tue, 18 Apr 2023 19:41:54 +0300
Subject: Strange Broken Pipe error from Varnish health checks
In-Reply-To: <CAOrPorn_8_Hj3igde=vthY1DisTrz09BCJvrfXEwiUAaOZ1Hkw@mail.gmail.com>
References: <CABd=SGWQaTntnh1prbnYWUF5L1QLK5ko++Nee3ApOsVe=oE05A@mail.gmail.com>
 <CAOrPorn9fn1dHP98CWUfiMMDMSyGr+64vHyu9QXgfU5SA4LxKA@mail.gmail.com>
 <CABd=SGXP+hZ481YmJ5q07Z1VTc3UwTHuH7hKMoN+BPq28KxRzg@mail.gmail.com>
 <CAOrPormEWzzPUxJqMMuHtD29A13tUbahLdPZ1P4UJidNbQg3+g@mail.gmail.com>
 <CABd=SGWDSKkRqAVAWOZBNR+knnaJND9=F0JLx31f=qeBQgf5AA@mail.gmail.com>
 <CAOrPorn_8_Hj3igde=vthY1DisTrz09BCJvrfXEwiUAaOZ1Hkw@mail.gmail.com>
Message-ID: <CABd=SGUeuP-r6iMFgJc6qGSuKSiLsTtRSMGTiHe8595umCaOtw@mail.gmail.com>

Hi,

Attached is the packet capture for the health check

Please check and advise.


?n lun., 17 apr. 2023 la 19:15, Guillaume Quintard <
guillaume.quintard at gmail.com> a scris:

> That code hasn't moved in a while, so I'd be surprised to see a bug there,
> but that's always possible.
> Any chance you could get a tcpdump of a probe request (from connection to
> disconnection) so we can see what's going on?
> --
> Guillaume Quintard
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/varnish-misc/attachments/20230418/a2b0bbbf/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: capture.pcap
Type: application/octet-stream
Size: 1446 bytes
Desc: not available
URL: <https://www.varnish-cache.org/lists/pipermail/varnish-misc/attachments/20230418/a2b0bbbf/attachment-0001.obj>

From guillaume.quintard at gmail.com  Tue Apr 18 18:26:34 2023
From: guillaume.quintard at gmail.com (Guillaume Quintard)
Date: Tue, 18 Apr 2023 11:26:34 -0700
Subject: Strange Broken Pipe error from Varnish health checks
In-Reply-To: <CABd=SGUeuP-r6iMFgJc6qGSuKSiLsTtRSMGTiHe8595umCaOtw@mail.gmail.com>
References: <CABd=SGWQaTntnh1prbnYWUF5L1QLK5ko++Nee3ApOsVe=oE05A@mail.gmail.com>
 <CAOrPorn9fn1dHP98CWUfiMMDMSyGr+64vHyu9QXgfU5SA4LxKA@mail.gmail.com>
 <CABd=SGXP+hZ481YmJ5q07Z1VTc3UwTHuH7hKMoN+BPq28KxRzg@mail.gmail.com>
 <CAOrPormEWzzPUxJqMMuHtD29A13tUbahLdPZ1P4UJidNbQg3+g@mail.gmail.com>
 <CABd=SGWDSKkRqAVAWOZBNR+knnaJND9=F0JLx31f=qeBQgf5AA@mail.gmail.com>
 <CAOrPorn_8_Hj3igde=vthY1DisTrz09BCJvrfXEwiUAaOZ1Hkw@mail.gmail.com>
 <CABd=SGUeuP-r6iMFgJc6qGSuKSiLsTtRSMGTiHe8595umCaOtw@mail.gmail.com>
Message-ID: <CAOrPor=pwoxfXeNfY62MWRABXmvUgGRDuW4jFhFmm_9SOdNKGA@mail.gmail.com>

Hi George,

That pcap only contains HTTP info, it would be super useful to have the TCP
packets, (SYN/ACK/FIN) to see who closes the connection on whom.
-- 
Guillaume Quintard


On Tue, Apr 18, 2023 at 9:43?AM George <izghitu at gmail.com> wrote:

> Hi,
>
> Attached is the packet capture for the health check
>
> Please check and advise.
>
>
> ?n lun., 17 apr. 2023 la 19:15, Guillaume Quintard <
> guillaume.quintard at gmail.com> a scris:
>
>> That code hasn't moved in a while, so I'd be surprised to see a bug
>> there, but that's always possible.
>> Any chance you could get a tcpdump of a probe request (from connection to
>> disconnection) so we can see what's going on?
>> --
>> Guillaume Quintard
>>
>> _______________________________________________
> varnish-misc mailing list
> varnish-misc at varnish-cache.org
> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/varnish-misc/attachments/20230418/a591653c/attachment.html>

From batanun at hotmail.com  Wed Apr 19 08:34:21 2023
From: batanun at hotmail.com (Batanun B)
Date: Wed, 19 Apr 2023 08:34:21 +0000
Subject: Possible to disable/inactivate a backend using VCL?
In-Reply-To: <CAOrPorkq4h5qRigFphQ5i5ckgoyx=_LvS1CWob2kPzHFdar1LQ@mail.gmail.com>
References: <PR3P190MB0922E70F10984632A678E67AAC999@PR3P190MB0922.EURP190.PROD.OUTLOOK.COM>
 <CAOrPorkq4h5qRigFphQ5i5ckgoyx=_LvS1CWob2kPzHFdar1LQ@mail.gmail.com>
Message-ID: <PR3P190MB0922DBF2C51ECB1A0C9E0599AC629@PR3P190MB0922.EURP190.PROD.OUTLOOK.COM>

Hi Guillaume,

> I'm curious, if it's completely deactivated what's the benefit of having it in the vcl?

It is only intended to be deactivated in production (until we go live). Our test and staging environments have the backend active.

> if (false) {
> ??set req.backend_hint = you_deactivated_backend;
> }

Thanks, I will test this.
My current prod-specific setup for this backend looks like this:

backend theBackend {
    .host = "localhost";
    .port = "9999";
    .probe = {
        .interval = 1h;
    }
}

This seems to be working when testing it locally. It also solves the problem of having to assign some arbitrary ip or hostname (the actual backend host for this service hasn't been created in production yet, since we are several months away from go live), which actually was our main problem. What do you think about this approach instead? Preferably this would be a built in feature in Varnish, with a setting "disabled = true" or similar in the backend definition, and then it would not require any host or ip to be configured.

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/varnish-misc/attachments/20230419/c7bb3240/attachment.html>

From batanun at hotmail.com  Wed Apr 19 08:48:02 2023
From: batanun at hotmail.com (Batanun B)
Date: Wed, 19 Apr 2023 08:48:02 +0000
Subject: Varnish won't start because backend host resolves to too many
 addresses, but they are all identical IPs
Message-ID: <PR3P190MB0922CB2EA0ECA3990F9946BFAC629@PR3P190MB0922.EURP190.PROD.OUTLOOK.COM>

All of the sudden Varnish fails to start in my development environment, and gives me the following error message:

Message from VCC-compiler:
Backend host "redacted-hostname": resolves to too many addresses.
Only one IPv4 and one IPv6 are allowed.
Please specify which exact address you want to use, we found all of these:
     555.123.123.3:80
     555.123.123.3:80
     555.123.123.3:80

I have changed the hostname and the IP above to not expose our server, but all three IP numbers are 100% identical. Shouldn't Varnish be able to figure out that in that case it can just choose any one and it will work as expected? It really should remove duplicates, and only if there are more than one non-duplicate IP then it should fail.

The problem is that the backend host is a so called "app service" in Microsoft Azure, which is basically a platform as a service (PaaS), where Microsoft handles the networking including the domain name (no user access it directly). I have no idea why it suddenly resolves to multiple duplicate IPs.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/varnish-misc/attachments/20230419/c1b63d53/attachment-0001.html>

From batanun at hotmail.com  Wed Apr 19 09:26:36 2023
From: batanun at hotmail.com (Batanun B)
Date: Wed, 19 Apr 2023 09:26:36 +0000
Subject: Strange Broken Pipe error from Varnish health checks
In-Reply-To: <CABd=SGWQaTntnh1prbnYWUF5L1QLK5ko++Nee3ApOsVe=oE05A@mail.gmail.com>
References: <CABd=SGWQaTntnh1prbnYWUF5L1QLK5ko++Nee3ApOsVe=oE05A@mail.gmail.com>
Message-ID: <PR3P190MB09227C1DEAC23D49A566115AAC629@PR3P190MB0922.EURP190.PROD.OUTLOOK.COM>

Couldn't a HEAD request solve this? Then nginx wouldn't bother with the body at all, right?

This is what we do with our health checks. For example:

backend someBackend {
    .host = "[redacted]";
    .port = "80";
    .probe = {
        .interval = 9s;
        .request =
            "HEAD /healthcheck HTTP/1.1"
            "Host: [redacted]"
            "User-Agent: varnish-health-probe"
            "Connection: Close"
            "Accept: */*";
    }
}
________________________________
From: varnish-misc <varnish-misc-bounces+batanun=hotmail.com at varnish-cache.org> on behalf of George <izghitu at gmail.com>
Sent: Monday, April 17, 2023 10:21 AM
To: varnish-misc at varnish-cache.org <varnish-misc at varnish-cache.org>
Subject: Strange Broken Pipe error from Varnish health checks

Hi,

I have a Varnish/nginx cluster running with varnish-7.1.2-1.el7.x86_64 on CentOS 7.

The issue I am having comes from the varnish health checks. I am getting a "broken pipe" error in the nginx error log at random times like below:
Apr 10 17:32:46 VARNISH-MASTER nginx_varnish_error: 2023/04/10 17:32:46 [info] 17808#17808: *67626636 writev() failed (32: Broken pipe), client: unix:, server: _, request: "GET /varnish_check HTTP/1.1", host: "0.0.0.0"

The strange thing is that this error appears only when Varnish performs the health checks. I have other scripts doing it(nagios, curl, wget, AWS ELB) but those do not show any errors. In addition to this Varnish and nginx where the health checks occur are on the same server and it makes no difference if I use a TCP connection or socket based one.

Below are the varnish vcl and nginx locations for the health checks:
backend nginx_varnish {
                   .path = "/run/nginx/nginx.sock";
                   .first_byte_timeout = 600s;
                   .probe = health;
        }

location = /varnish_check {
                keepalive_timeout 305;
                return 200 'Varnish Check';
                access_log /var/log/nginx/varnish_check.log main;
                error_log /var/log/nginx/varnish_check_errors.log debug;
                error_log syslog:server=unix:/run/nginx_log.in.sock,facility=local1,tag=nginx_varnish_error,nohostname info;
        }

Are there any docs I can read about how exactly varnish performs the health checks and what internal processes are involved?
Did anyone happen to have similar issues? This is not causing any operational problems for the cluster but it is just something that I want to determine why it is happening because it just should not be happening.

Please help
THanks in advance.

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/varnish-misc/attachments/20230419/f5c37e28/attachment.html>

From guillaume.quintard at gmail.com  Wed Apr 19 14:22:15 2023
From: guillaume.quintard at gmail.com (Guillaume Quintard)
Date: Wed, 19 Apr 2023 07:22:15 -0700
Subject: Possible to disable/inactivate a backend using VCL?
In-Reply-To: <PR3P190MB0922DBF2C51ECB1A0C9E0599AC629@PR3P190MB0922.EURP190.PROD.OUTLOOK.COM>
References: <PR3P190MB0922E70F10984632A678E67AAC999@PR3P190MB0922.EURP190.PROD.OUTLOOK.COM>
 <CAOrPorkq4h5qRigFphQ5i5ckgoyx=_LvS1CWob2kPzHFdar1LQ@mail.gmail.com>
 <PR3P190MB0922DBF2C51ECB1A0C9E0599AC629@PR3P190MB0922.EURP190.PROD.OUTLOOK.COM>
Message-ID: <CAOrPorm+te4AogMvfZTnD7kVDJQEosqDv5tZF0630yTWAipNEA@mail.gmail.com>

Thank, I think I get it now. How about:

backend theBackend none;

Here's the relevant documentation:
https://varnish-cache.org/docs/trunk/users-guide/vcl-backends.html#the-none-backend
It was added in 6.4.

Hope that helps.

-- 
Guillaume Quintard


On Wed, Apr 19, 2023 at 1:36?AM Batanun B <batanun at hotmail.com> wrote:

> Hi Guillaume,
>
> > I'm curious, if it's completely deactivated what's the benefit of having
> it in the vcl?
>
> It is only intended to be deactivated in production (until we go live).
> Our test and staging environments have the backend active.
>
> > if (false) {
> > set req.backend_hint = you_deactivated_backend;
> > }
>
> Thanks, I will test this.
> My current prod-specific setup for this backend looks like this:
>
> backend theBackend {
>     .host = "localhost";
>     .port = "9999";
>     .probe = {
>         .interval = 1h;
>     }
> }
>
> This seems to be working when testing it locally. It also solves the
> problem of having to assign some arbitrary ip or hostname (the actual
> backend host for this service hasn't been created in production yet, since
> we are several months away from go live), which actually was our main
> problem. What do you think about this approach instead? Preferably this
> would be a built in feature in Varnish, with a setting "disabled = true" or
> similar in the backend definition, and then it would not require any host
> or ip to be configured.
>
> _______________________________________________
> varnish-misc mailing list
> varnish-misc at varnish-cache.org
> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/varnish-misc/attachments/20230419/0aac69a8/attachment.html>

From guillaume.quintard at gmail.com  Wed Apr 19 14:42:23 2023
From: guillaume.quintard at gmail.com (Guillaume Quintard)
Date: Wed, 19 Apr 2023 07:42:23 -0700
Subject: Varnish won't start because backend host resolves to too many
 addresses, but they are all identical IPs
In-Reply-To: <PR3P190MB0922CB2EA0ECA3990F9946BFAC629@PR3P190MB0922.EURP190.PROD.OUTLOOK.COM>
References: <PR3P190MB0922CB2EA0ECA3990F9946BFAC629@PR3P190MB0922.EURP190.PROD.OUTLOOK.COM>
Message-ID: <CAOrPormiRRSBqX6g3ELTPyB_7GAvaP-RLSBmyjJSH32hMh=UgA@mail.gmail.com>

The fact the IPs are identical is weird, but I wouldn't be surprised if the
dns entry actually contained 3 identical IPs.

> Shouldn't Varnish be able to figure out that in that case it can just
choose any one and it will work as expected?

Shouldn't your DNS entries be clean? ;-)

Honestly, if the IP(s) behind the service name is liable to change, you
shouldn't use a dynamic backend because Varnish resolves the IP when the
VCL is loaded, so if the IP changes behind your back, Varnish won't follow
it, and you'll be screwed.
Instead, you should use dynamic backends, of which there are a handful:
- dynamic <https://github.com/nigoroll/libvmod-dynamic>, by UPLEX: it's
been around for ages, it's battle-tested, and it's included in the oficial
Varnish Docker image <https://hub.docker.com/_/varnish>
- udo+activedns
<https://docs.varnish-software.com/varnish-enterprise/vmods/udo/#subscribe>,
by Varnish Software: the design is slightly different and allows you to
specify pretty much any load-balancing policy you might need. You'll need a
subscription but you'll get excellent support (disclaimer, I'm an ex
employee)
- reqwest
<https://github.com/gquintard/vmod_reqwest#backend-https-following-up-to-5-redirect-hops-and-brotli-auto-decompression>,
by yours truly: the interface focuses on providing a simple experience and
a few bells and whistles (HTTPS, HTTP2, brotli, following redirects)

As you can see, the static backend's reluctance to fully handle DNS has
been a fertile ground for vmods :-)

-- 
Guillaume Quintard


On Wed, Apr 19, 2023 at 1:49?AM Batanun B <batanun at hotmail.com> wrote:

> All of the sudden Varnish fails to start in my development environment,
> and gives me the following error message:
>
> Message from VCC-compiler:
> Backend host "redacted-hostname": resolves to too many addresses.
> Only one IPv4 and one IPv6 are allowed.
> Please specify which exact address you want to use, we found all of these:
>      555.123.123.3:80
>      555.123.123.3:80
>      555.123.123.3:80
>
> I have changed the hostname and the IP above to not expose our server, but
> all three IP numbers are 100% identical. Shouldn't Varnish be able to
> figure out that in that case it can just choose any one and it will work as
> expected? It really should remove duplicates, and only if there are more
> than one non-duplicate IP then it should fail.
>
> The problem is that the backend host is a so called "app service" in
> Microsoft Azure, which is basically a platform as a service (PaaS), where
> Microsoft handles the networking including the domain name (no user access
> it directly). I have no idea why it suddenly resolves to multiple duplicate
> IPs.
> _______________________________________________
> varnish-misc mailing list
> varnish-misc at varnish-cache.org
> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/varnish-misc/attachments/20230419/580ec47d/attachment-0001.html>

From dridi at varni.sh  Wed Apr 19 15:38:29 2023
From: dridi at varni.sh (Dridi Boukelmoune)
Date: Wed, 19 Apr 2023 15:38:29 +0000
Subject: Varnish won't start because backend host resolves to too many
 addresses, but they are all identical IPs
In-Reply-To: <CAOrPormiRRSBqX6g3ELTPyB_7GAvaP-RLSBmyjJSH32hMh=UgA@mail.gmail.com>
References: <PR3P190MB0922CB2EA0ECA3990F9946BFAC629@PR3P190MB0922.EURP190.PROD.OUTLOOK.COM>
 <CAOrPormiRRSBqX6g3ELTPyB_7GAvaP-RLSBmyjJSH32hMh=UgA@mail.gmail.com>
Message-ID: <CABoVN9B6faeK_f0wLzHMKpCW9Oct0cpKbakppJw1Gnj=MDWRiQ@mail.gmail.com>

On Wed, Apr 19, 2023 at 2:44?PM Guillaume Quintard
<guillaume.quintard at gmail.com> wrote:
>
> The fact the IPs are identical is weird, but I wouldn't be surprised if the dns entry actually contained 3 identical IPs.
>
> > Shouldn't Varnish be able to figure out that in that case it can just choose any one and it will work as expected?
>
> Shouldn't your DNS entries be clean? ;-)

It should, but Varnish could probably make an effort here.

From batanun at hotmail.com  Wed Apr 19 15:43:07 2023
From: batanun at hotmail.com (Batanun B)
Date: Wed, 19 Apr 2023 15:43:07 +0000
Subject: Possible to disable/inactivate a backend using VCL?
In-Reply-To: <CAOrPorm+te4AogMvfZTnD7kVDJQEosqDv5tZF0630yTWAipNEA@mail.gmail.com>
References: <PR3P190MB0922E70F10984632A678E67AAC999@PR3P190MB0922.EURP190.PROD.OUTLOOK.COM>
 <CAOrPorkq4h5qRigFphQ5i5ckgoyx=_LvS1CWob2kPzHFdar1LQ@mail.gmail.com>
 <PR3P190MB0922DBF2C51ECB1A0C9E0599AC629@PR3P190MB0922.EURP190.PROD.OUTLOOK.COM>
 <CAOrPorm+te4AogMvfZTnD7kVDJQEosqDv5tZF0630yTWAipNEA@mail.gmail.com>
Message-ID: <PR3P190MB092274A64AFEA7ADABD39783AC629@PR3P190MB0922.EURP190.PROD.OUTLOOK.COM>

> backend theBackend none;
> Here's the relevant documentation: https://varnish-cache.org/docs/trunk/users-guide/vcl-backends.html#the-none-backend
> It was added in 6.4.

Look like exactly what we need! Sadly we are "stuck" on 6.0 until the next LTS version comes. So I think that until then I will use our poor man version of the "none" backend, ie pointing to localhost with an port number that won't give a response.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/varnish-misc/attachments/20230419/9490b496/attachment.html>

From batanun at hotmail.com  Wed Apr 19 16:02:08 2023
From: batanun at hotmail.com (Batanun B)
Date: Wed, 19 Apr 2023 16:02:08 +0000
Subject: Varnish won't start because backend host resolves to too many
 addresses, but they are all identical IPs
In-Reply-To: <CAOrPormiRRSBqX6g3ELTPyB_7GAvaP-RLSBmyjJSH32hMh=UgA@mail.gmail.com>
References: <PR3P190MB0922CB2EA0ECA3990F9946BFAC629@PR3P190MB0922.EURP190.PROD.OUTLOOK.COM>
 <CAOrPormiRRSBqX6g3ELTPyB_7GAvaP-RLSBmyjJSH32hMh=UgA@mail.gmail.com>
Message-ID: <PR3P190MB0922FDF413E16D1ADCCB7678AC629@PR3P190MB0922.EURP190.PROD.OUTLOOK.COM>

> Shouldn't your DNS entries be clean? ;-)

Preferably, but I blame Microsoft here ??

The problem went away by itself when I tried starting again like half an hour later or so, so I guess it was a temporary glitch in the matrix.

As far as I understand it, the IPs of these machines only change if they are deleted and created again. We do it occasionally in test/staging, and there we can live with Varnish needing to be restarted. In production we don't really delete them once they are properly setup, unless there is some major problem and then a restart of the load balanced varnish servers should not be a concern.

Thanks for your vmod suggestions! I will check them out. The dynamic one seems like the only one that supports community edition LTS 6.0. The documentation seems a bit lacking (no full VCL example), but I guess I could use their test cases as examples.

________________________________
From: Guillaume Quintard <guillaume.quintard at gmail.com>
Sent: Wednesday, April 19, 2023 4:42 PM
To: Batanun B <batanun at hotmail.com>
Cc: varnish-misc at varnish-cache.org <varnish-misc at varnish-cache.org>
Subject: Re: Varnish won't start because backend host resolves to too many addresses, but they are all identical IPs

The fact the IPs are identical is weird, but I wouldn't be surprised if the dns entry actually contained 3 identical IPs.

> Shouldn't Varnish be able to figure out that in that case it can just choose any one and it will work as expected?

Shouldn't your DNS entries be clean? ;-)

Honestly, if the IP(s) behind the service name is liable to change, you shouldn't use a dynamic backend because Varnish resolves the IP when the VCL is loaded, so if the IP changes behind your back, Varnish won't follow it, and you'll be screwed.
Instead, you should use dynamic backends, of which there are a handful:
- dynamic<https://github.com/nigoroll/libvmod-dynamic>, by UPLEX: it's been around for ages, it's battle-tested, and it's included in the oficial Varnish Docker image<https://hub.docker.com/_/varnish>
- udo+activedns<https://docs.varnish-software.com/varnish-enterprise/vmods/udo/#subscribe>, by Varnish Software: the design is slightly different and allows you to specify pretty much any load-balancing policy you might need. You'll need a subscription but you'll get excellent support (disclaimer, I'm an ex employee)
- reqwest<https://github.com/gquintard/vmod_reqwest#backend-https-following-up-to-5-redirect-hops-and-brotli-auto-decompression>, by yours truly: the interface focuses on providing a simple experience and a few bells and whistles (HTTPS, HTTP2, brotli, following redirects)

As you can see, the static backend's reluctance to fully handle DNS has been a fertile ground for vmods :-)

--
Guillaume Quintard


On Wed, Apr 19, 2023 at 1:49?AM Batanun B <batanun at hotmail.com<mailto:batanun at hotmail.com>> wrote:
All of the sudden Varnish fails to start in my development environment, and gives me the following error message:

Message from VCC-compiler:
Backend host "redacted-hostname": resolves to too many addresses.
Only one IPv4 and one IPv6 are allowed.
Please specify which exact address you want to use, we found all of these:
     555.123.123.3:80
     555.123.123.3:80
     555.123.123.3:80

I have changed the hostname and the IP above to not expose our server, but all three IP numbers are 100% identical. Shouldn't Varnish be able to figure out that in that case it can just choose any one and it will work as expected? It really should remove duplicates, and only if there are more than one non-duplicate IP then it should fail.

The problem is that the backend host is a so called "app service" in Microsoft Azure, which is basically a platform as a service (PaaS), where Microsoft handles the networking including the domain name (no user access it directly). I have no idea why it suddenly resolves to multiple duplicate IPs.
_______________________________________________
varnish-misc mailing list
varnish-misc at varnish-cache.org<mailto:varnish-misc at varnish-cache.org>
https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/varnish-misc/attachments/20230419/3d88fcc1/attachment-0001.html>

From guillaume.quintard at gmail.com  Wed Apr 19 16:20:49 2023
From: guillaume.quintard at gmail.com (Guillaume Quintard)
Date: Wed, 19 Apr 2023 09:20:49 -0700
Subject: Varnish won't start because backend host resolves to too many
 addresses, but they are all identical IPs
In-Reply-To: <PR3P190MB0922FDF413E16D1ADCCB7678AC629@PR3P190MB0922.EURP190.PROD.OUTLOOK.COM>
References: <PR3P190MB0922CB2EA0ECA3990F9946BFAC629@PR3P190MB0922.EURP190.PROD.OUTLOOK.COM>
 <CAOrPormiRRSBqX6g3ELTPyB_7GAvaP-RLSBmyjJSH32hMh=UgA@mail.gmail.com>
 <PR3P190MB0922FDF413E16D1ADCCB7678AC629@PR3P190MB0922.EURP190.PROD.OUTLOOK.COM>
Message-ID: <CAOrPormaWas+6vzJr7HkEghZX0rpbFWDx9L8Q2v+KC3o0p5hCA@mail.gmail.com>

> The documentation seems a bit lacking (no full VCL example), but I guess
I could use their test cases as examples.

https://github.com/nigoroll/libvmod-dynamic/blob/master/src/vmod_dynamic.vcc#L538-L583
maybe?
I'm sure Nils will pipe up here if you need help, and if you want more
synchronous assistance, there's always the discord channel
<https://varnish-cache.org/support/>.

> The dynamic one seems like the only one that supports community edition
LTS 6.0.

Yes, of the three, that's the only one that will support that one (VS is
focused on the Enterprise version, and I lack the time to port vmods on 6.0
(but I'll welcome the help)).

Cheers,

-- 
Guillaume Quintard


On Wed, Apr 19, 2023 at 9:02?AM Batanun B <batanun at hotmail.com> wrote:

> > Shouldn't your DNS entries be clean? ;-)
>
> Preferably, but I blame Microsoft here ??
>
> The problem went away by itself when I tried starting again like half an
> hour later or so, so I guess it was a temporary glitch in the matrix.
>
> As far as I understand it, the IPs of these machines only change if they
> are deleted and created again. We do it occasionally in test/staging, and
> there we can live with Varnish needing to be restarted. In production we
> don't really delete them once they are properly setup, unless there is some
> major problem and then a restart of the load balanced varnish servers
> should not be a concern.
>
> Thanks for your vmod suggestions! I will check them out. The dynamic one
> seems like the only one that supports community edition LTS 6.0. The
> documentation seems a bit lacking (no full VCL example), but I guess I
> could use their test cases as examples.
>
> ------------------------------
> *From:* Guillaume Quintard <guillaume.quintard at gmail.com>
> *Sent:* Wednesday, April 19, 2023 4:42 PM
> *To:* Batanun B <batanun at hotmail.com>
> *Cc:* varnish-misc at varnish-cache.org <varnish-misc at varnish-cache.org>
> *Subject:* Re: Varnish won't start because backend host resolves to too
> many addresses, but they are all identical IPs
>
> The fact the IPs are identical is weird, but I wouldn't be surprised if
> the dns entry actually contained 3 identical IPs.
>
> > Shouldn't Varnish be able to figure out that in that case it can just
> choose any one and it will work as expected?
>
> Shouldn't your DNS entries be clean? ;-)
>
> Honestly, if the IP(s) behind the service name is liable to change, you
> shouldn't use a dynamic backend because Varnish resolves the IP when the
> VCL is loaded, so if the IP changes behind your back, Varnish won't follow
> it, and you'll be screwed.
> Instead, you should use dynamic backends, of which there are a handful:
> - dynamic <https://github.com/nigoroll/libvmod-dynamic>, by UPLEX: it's
> been around for ages, it's battle-tested, and it's included in the oficial
> Varnish Docker image <https://hub.docker.com/_/varnish>
> - udo+activedns
> <https://docs.varnish-software.com/varnish-enterprise/vmods/udo/#subscribe>,
> by Varnish Software: the design is slightly different and allows you to
> specify pretty much any load-balancing policy you might need. You'll need a
> subscription but you'll get excellent support (disclaimer, I'm an ex
> employee)
> - reqwest
> <https://github.com/gquintard/vmod_reqwest#backend-https-following-up-to-5-redirect-hops-and-brotli-auto-decompression>,
> by yours truly: the interface focuses on providing a simple experience and
> a few bells and whistles (HTTPS, HTTP2, brotli, following redirects)
>
> As you can see, the static backend's reluctance to fully handle DNS has
> been a fertile ground for vmods :-)
>
> --
> Guillaume Quintard
>
>
> On Wed, Apr 19, 2023 at 1:49?AM Batanun B <batanun at hotmail.com> wrote:
>
> All of the sudden Varnish fails to start in my development environment,
> and gives me the following error message:
>
> Message from VCC-compiler:
> Backend host "redacted-hostname": resolves to too many addresses.
> Only one IPv4 and one IPv6 are allowed.
> Please specify which exact address you want to use, we found all of these:
>      555.123.123.3:80
>      555.123.123.3:80
>      555.123.123.3:80
>
> I have changed the hostname and the IP above to not expose our server, but
> all three IP numbers are 100% identical. Shouldn't Varnish be able to
> figure out that in that case it can just choose any one and it will work as
> expected? It really should remove duplicates, and only if there are more
> than one non-duplicate IP then it should fail.
>
> The problem is that the backend host is a so called "app service" in
> Microsoft Azure, which is basically a platform as a service (PaaS), where
> Microsoft handles the networking including the domain name (no user access
> it directly). I have no idea why it suddenly resolves to multiple duplicate
> IPs.
> _______________________________________________
> varnish-misc mailing list
> varnish-misc at varnish-cache.org
> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/varnish-misc/attachments/20230419/c21f542a/attachment.html>

From batanun at hotmail.com  Wed Apr 19 16:22:49 2023
From: batanun at hotmail.com (Batanun B)
Date: Wed, 19 Apr 2023 16:22:49 +0000
Subject: Is there any "try catch" functionality in VCL? If not, how to handle
 runtime errors in vcl_init?
Message-ID: <PR3P190MB0922E8B6ED69B6B050CFEAF9AC629@PR3P190MB0922.EURP190.PROD.OUTLOOK.COM>

Hi,

We use the vmod crypto to verify cryptographic signatures for some of our traffic. When testing, the public key was hard coded in the VCL, but before we start using this feature in production we will switch to reading the public key from a file on disk. This file is generated on server startup, by fetching it from an Azure keyvault.

Now, the problem I'm picturing here is that this fetching of the public key can fail, or the key can be corrupt or empty, maybe by user error. Or the key could be valid, but the format of the key happens to be unsupported by the vmod crypto. So, even if we do our best to validate the key, in theory it could pass all our tests but still fail when we give it to the vmod crypto. And if that happens, Varnish won't start because the vmod crypto is initiated with the public key in vcl_init, like this:

sub vcl_init {
  new cryptoVerifier = crypto.verifier(sha256, std.fileread("/path/to/public.key"));
}

What I would prefer to happen if the key is rejected, is that vcl_init goes through without failure, and then the requests that use the cryptoVerifier will fail, but all other traffic (like 99%) still works. Can we achieve this somehow? Like some try-catch functionallity? If not, is there some other way to handle this that doesn't cause Varnish to die on startup?
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/varnish-misc/attachments/20230419/cf64d350/attachment-0001.html>

From dridi at varni.sh  Wed Apr 19 16:25:38 2023
From: dridi at varni.sh (Dridi Boukelmoune)
Date: Wed, 19 Apr 2023 16:25:38 +0000
Subject: Possible to disable/inactivate a backend using VCL?
In-Reply-To: <PR3P190MB092274A64AFEA7ADABD39783AC629@PR3P190MB0922.EURP190.PROD.OUTLOOK.COM>
References: <PR3P190MB0922E70F10984632A678E67AAC999@PR3P190MB0922.EURP190.PROD.OUTLOOK.COM>
 <CAOrPorkq4h5qRigFphQ5i5ckgoyx=_LvS1CWob2kPzHFdar1LQ@mail.gmail.com>
 <PR3P190MB0922DBF2C51ECB1A0C9E0599AC629@PR3P190MB0922.EURP190.PROD.OUTLOOK.COM>
 <CAOrPorm+te4AogMvfZTnD7kVDJQEosqDv5tZF0630yTWAipNEA@mail.gmail.com>
 <PR3P190MB092274A64AFEA7ADABD39783AC629@PR3P190MB0922.EURP190.PROD.OUTLOOK.COM>
Message-ID: <CABoVN9D6uXZqRQj+6-+UzogGKjqrTd4Z1QhdW836YtOdHXB+UQ@mail.gmail.com>

On Wed, Apr 19, 2023 at 3:45?PM Batanun B <batanun at hotmail.com> wrote:
>
> > backend theBackend none;
> > Here's the relevant documentation: https://varnish-cache.org/docs/trunk/users-guide/vcl-backends.html#the-none-backend
> > It was added in 6.4.
>
> Look like exactly what we need! Sadly we are "stuck" on 6.0 until the next LTS version comes. So I think that until then I will use our poor man version of the "none" backend, ie pointing to localhost with an port number that won't give a response.

It was back-ported to 6.0, which is not an LTS branch limited to bug fixes ;)

https://varnish-cache.org/docs/6.0/users-guide/vcl-backends.html

Dridi

From dridi at varni.sh  Wed Apr 19 16:29:08 2023
From: dridi at varni.sh (Dridi Boukelmoune)
Date: Wed, 19 Apr 2023 16:29:08 +0000
Subject: Is there any "try catch" functionality in VCL? If not, how to
 handle runtime errors in vcl_init?
In-Reply-To: <PR3P190MB0922E8B6ED69B6B050CFEAF9AC629@PR3P190MB0922.EURP190.PROD.OUTLOOK.COM>
References: <PR3P190MB0922E8B6ED69B6B050CFEAF9AC629@PR3P190MB0922.EURP190.PROD.OUTLOOK.COM>
Message-ID: <CABoVN9BAdy6nj_ErYr712LkLT-AXwvTq393HaEunavTNeFSvBg@mail.gmail.com>

On Wed, Apr 19, 2023 at 4:25?PM Batanun B <batanun at hotmail.com> wrote:
>
> Hi,
>
> We use the vmod crypto to verify cryptographic signatures for some of our traffic. When testing, the public key was hard coded in the VCL, but before we start using this feature in production we will switch to reading the public key from a file on disk. This file is generated on server startup, by fetching it from an Azure keyvault.
>
> Now, the problem I'm picturing here is that this fetching of the public key can fail, or the key can be corrupt or empty, maybe by user error. Or the key could be valid, but the format of the key happens to be unsupported by the vmod crypto. So, even if we do our best to validate the key, in theory it could pass all our tests but still fail when we give it to the vmod crypto. And if that happens, Varnish won't start because the vmod crypto is initiated with the public key in vcl_init, like this:
>
> sub vcl_init {
>   new cryptoVerifier = crypto.verifier(sha256, std.fileread("/path/to/public.key"));
> }
>
> What I would prefer to happen if the key is rejected, is that vcl_init goes through without failure, and then the requests that use the cryptoVerifier will fail, but all other traffic (like 99%) still works. Can we achieve this somehow? Like some try-catch functionallity? If not, is there some other way to handle this that doesn't cause Varnish to die on startup?

It's the VMOD author you should ask to have an option to ignore public
key errors.

This is a constructor, and even if we had a try-catch kind of
construct in the language, I don't think we would make this one
recoverable.

Dridi

From batanun at hotmail.com  Wed Apr 19 16:36:36 2023
From: batanun at hotmail.com (Batanun B)
Date: Wed, 19 Apr 2023 16:36:36 +0000
Subject: Possible to disable/inactivate a backend using VCL?
In-Reply-To: <CABoVN9D6uXZqRQj+6-+UzogGKjqrTd4Z1QhdW836YtOdHXB+UQ@mail.gmail.com>
References: <PR3P190MB0922E70F10984632A678E67AAC999@PR3P190MB0922.EURP190.PROD.OUTLOOK.COM>
 <CAOrPorkq4h5qRigFphQ5i5ckgoyx=_LvS1CWob2kPzHFdar1LQ@mail.gmail.com>
 <PR3P190MB0922DBF2C51ECB1A0C9E0599AC629@PR3P190MB0922.EURP190.PROD.OUTLOOK.COM>
 <CAOrPorm+te4AogMvfZTnD7kVDJQEosqDv5tZF0630yTWAipNEA@mail.gmail.com>
 <PR3P190MB092274A64AFEA7ADABD39783AC629@PR3P190MB0922.EURP190.PROD.OUTLOOK.COM>
 <CABoVN9D6uXZqRQj+6-+UzogGKjqrTd4Z1QhdW836YtOdHXB+UQ@mail.gmail.com>
Message-ID: <PR3P190MB092281D4FB74F13328DDB388AC629@PR3P190MB0922.EURP190.PROD.OUTLOOK.COM>

> It was back-ported to 6.0, which is not an LTS branch limited to bug fixes ;)
>
> https://varnish-cache.org/docs/6.0/users-guide/vcl-backends.html

Thanks! Wow, I can't believe that I could miss that. I thought I read that specific page, as well as searched on Google, but I guess I was too focused on looking for terms like "disabled" or "inactivated".

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/varnish-misc/attachments/20230419/69307870/attachment.html>

From batanun at hotmail.com  Wed Apr 19 16:45:01 2023
From: batanun at hotmail.com (Batanun B)
Date: Wed, 19 Apr 2023 16:45:01 +0000
Subject: Is there any "try catch" functionality in VCL? If not, how to
 handle runtime errors in vcl_init?
In-Reply-To: <CABoVN9BAdy6nj_ErYr712LkLT-AXwvTq393HaEunavTNeFSvBg@mail.gmail.com>
References: <PR3P190MB0922E8B6ED69B6B050CFEAF9AC629@PR3P190MB0922.EURP190.PROD.OUTLOOK.COM>
 <CABoVN9BAdy6nj_ErYr712LkLT-AXwvTq393HaEunavTNeFSvBg@mail.gmail.com>
Message-ID: <PR3P190MB09222AF6613C63EEE04706E7AC629@PR3P190MB0922.EURP190.PROD.OUTLOOK.COM>

> It's the VMOD author you should ask to have an option to ignore public
> key errors.

Well, I'm usually of the mindset that if a problem can be handled in a generic way by the language/platform/framework, then one should avoid requiring each and every custom vmod/plugin/library to handle it individually. And I'm also of the mindset that pretty much any non-trivial code can fail, and the calling code should be able to catch that if needed. :)

> This is a constructor, and even if we had a try-catch kind of
> construct in the language, I don't think we would make this one
> recoverable.

In my mind, with a try-catch I could handle it like this:

try {
  new cryptoVerifier = crypto.verifier(sha256, std.fileread("/path/to/public.key"));
} catch (error) {
  // log error...
  // then try with with a hard coded known safe key, but that will fail when checking the signature
  new cryptoVerifier = crypto.verifier(sha256, (sha256, {"
-----BEGIN PUBLIC KEY-----
...
-----END PUBLIC KEY-----
"});
}

With this approach, Varnish will start like normal. And the only requests failing will be the ones using the cryptoVerifier.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/varnish-misc/attachments/20230419/65637ffc/attachment-0001.html>

From batanun at hotmail.com  Wed Apr 19 16:48:32 2023
From: batanun at hotmail.com (Batanun B)
Date: Wed, 19 Apr 2023 16:48:32 +0000
Subject: Varnish won't start because backend host resolves to too many
 addresses, but they are all identical IPs
In-Reply-To: <CAOrPormaWas+6vzJr7HkEghZX0rpbFWDx9L8Q2v+KC3o0p5hCA@mail.gmail.com>
References: <PR3P190MB0922CB2EA0ECA3990F9946BFAC629@PR3P190MB0922.EURP190.PROD.OUTLOOK.COM>
 <CAOrPormiRRSBqX6g3ELTPyB_7GAvaP-RLSBmyjJSH32hMh=UgA@mail.gmail.com>
 <PR3P190MB0922FDF413E16D1ADCCB7678AC629@PR3P190MB0922.EURP190.PROD.OUTLOOK.COM>
 <CAOrPormaWas+6vzJr7HkEghZX0rpbFWDx9L8Q2v+KC3o0p5hCA@mail.gmail.com>
Message-ID: <PR3P190MB092264D0CE21EC94F8431643AC629@PR3P190MB0922.EURP190.PROD.OUTLOOK.COM>

> https://github.com/nigoroll/libvmod-dynamic/blob/master/src/vmod_dynamic.vcc#L538-L583
> maybe?
>

> I'm sure Nils will pipe up here if you need help, and if you want more
> synchronous assistance, there's always the discord channel
> <https://varnish-cache.org/support/>.

Thanks! :)

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/varnish-misc/attachments/20230419/f358dd39/attachment.html>

From batanun at hotmail.com  Wed Apr 19 17:04:18 2023
From: batanun at hotmail.com (Batanun B)
Date: Wed, 19 Apr 2023 17:04:18 +0000
Subject: Is there any "try catch" functionality in VCL? If not, how to
 handle runtime errors in vcl_init?
In-Reply-To: <PR3P190MB09222AF6613C63EEE04706E7AC629@PR3P190MB0922.EURP190.PROD.OUTLOOK.COM>
References: <PR3P190MB0922E8B6ED69B6B050CFEAF9AC629@PR3P190MB0922.EURP190.PROD.OUTLOOK.COM>
 <CABoVN9BAdy6nj_ErYr712LkLT-AXwvTq393HaEunavTNeFSvBg@mail.gmail.com>
 <PR3P190MB09222AF6613C63EEE04706E7AC629@PR3P190MB0922.EURP190.PROD.OUTLOOK.COM>
Message-ID: <PR3P190MB0922E8B3C69F625C2EC8ACF4AC629@PR3P190MB0922.EURP190.PROD.OUTLOOK.COM>

Just to explain my concern a bit. The worst case?scenario in production, that I very much would like to avoid, could look something like this:

1. Something happens with our public key, so that Varnish won't be able to start after getting the new faulty key. Already running servers will continue to run, but no new servers can be initiated.
2. Before we have been able to fix the problem with the public key, some other problem happens in Varnish, and all running Varnish servers dies.
3. We end up with no working Varnish servers running, and not being able to start any new ones.
4. All our websites are down.