From martynas at atomgraph.com  Sat Oct  2 22:14:18 2021
From: martynas at atomgraph.com (=?UTF-8?Q?Martynas_Jusevi=C4=8Dius?=)
Date: Sun, 3 Oct 2021 00:14:18 +0200
Subject: varnishadm exit code 200
Message-ID: <CAE35VmxO+eO_4jLyxEWu-pnDZX-9Gw_DfEh6X=n_00Wni068Gg@mail.gmail.com>

Hi,

We recently switched to the varnish:latest container and based an
unprivileged image on it (entrypoint runs as USER varnish):
https://github.com/AtomGraph/varnish/blob/official-image/Dockerfile

We noticed that our varnishadm commands started failing. More specifically:

root at dc17c642d39a:/etc/varnish# varnishadm "ban req.url ~ /"
200

As I understand it's a 200 exit code, which means varnishadm failed:
https://varnish-cache.org/docs/5.1/reference/varnishadm.html#exit-status

What does 200 mean exactly? I couldn't find any code list.
My guess is that this has to do with the unprivileged varnish user,
but I'm not sure what it takes to fix it.


Martynas

From martynas at atomgraph.com  Sat Oct  2 22:23:33 2021
From: martynas at atomgraph.com (=?UTF-8?Q?Martynas_Jusevi=C4=8Dius?=)
Date: Sun, 3 Oct 2021 00:23:33 +0200
Subject: varnishadm exit code 200
In-Reply-To: <CAE35VmxO+eO_4jLyxEWu-pnDZX-9Gw_DfEh6X=n_00Wni068Gg@mail.gmail.com>
References: <CAE35VmxO+eO_4jLyxEWu-pnDZX-9Gw_DfEh6X=n_00Wni068Gg@mail.gmail.com>
Message-ID: <CAE35Vmw5gSCSEwvSPDcZ_HgoAYu9WJUWgozBAkYpFHCnJOAOUg@mail.gmail.com>

Actually it does not seem to be the exit code. I tried checking and it
looks like the exit code is 0:

root at dc17c642d39a:/etc/varnish# varnishadm "ban req.url ~ /"
200

root at dc17c642d39a:/etc/varnish# test $? -eq 0 || echo "Error"
root at dc17c642d39a:/etc/varnish#

So where is that "200" coming from?

On Sun, Oct 3, 2021 at 12:14 AM Martynas Jusevi?ius
<martynas at atomgraph.com> wrote:
>
> Hi,
>
> We recently switched to the varnish:latest container and based an
> unprivileged image on it (entrypoint runs as USER varnish):
> https://github.com/AtomGraph/varnish/blob/official-image/Dockerfile
>
> We noticed that our varnishadm commands started failing. More specifically:
>
> root at dc17c642d39a:/etc/varnish# varnishadm "ban req.url ~ /"
> 200
>
> As I understand it's a 200 exit code, which means varnishadm failed:
> https://varnish-cache.org/docs/5.1/reference/varnishadm.html#exit-status
>
> What does 200 mean exactly? I couldn't find any code list.
> My guess is that this has to do with the unprivileged varnish user,
> but I'm not sure what it takes to fix it.
>
>
> Martynas

From guillaume.quintard at gmail.com  Sat Oct  2 22:39:07 2021
From: guillaume.quintard at gmail.com (Guillaume Quintard)
Date: Sat, 2 Oct 2021 15:39:07 -0700
Subject: varnishadm exit code 200
In-Reply-To: <CAE35Vmw5gSCSEwvSPDcZ_HgoAYu9WJUWgozBAkYpFHCnJOAOUg@mail.gmail.com>
References: <CAE35VmxO+eO_4jLyxEWu-pnDZX-9Gw_DfEh6X=n_00Wni068Gg@mail.gmail.com>
 <CAE35Vmw5gSCSEwvSPDcZ_HgoAYu9WJUWgozBAkYpFHCnJOAOUg@mail.gmail.com>
Message-ID: <CAOrPormKS=R0FC94ScLHt23zF5n8CFH9KiHiaqbUNFWS=0zA6g@mail.gmail.com>

200 means the command passed to varnish, via varnishadm, succeeded. What
makes you think it failed?

On Sat, Oct 2, 2021, 15:24 Martynas Jusevi?ius <martynas at atomgraph.com>
wrote:

> Actually it does not seem to be the exit code. I tried checking and it
> looks like the exit code is 0:
>
> root at dc17c642d39a:/etc/varnish# varnishadm "ban req.url ~ /"
> 200
>
> root at dc17c642d39a:/etc/varnish# test $? -eq 0 || echo "Error"
> root at dc17c642d39a:/etc/varnish#
>
> So where is that "200" coming from?
>
> On Sun, Oct 3, 2021 at 12:14 AM Martynas Jusevi?ius
> <martynas at atomgraph.com> wrote:
> >
> > Hi,
> >
> > We recently switched to the varnish:latest container and based an
> > unprivileged image on it (entrypoint runs as USER varnish):
> > https://github.com/AtomGraph/varnish/blob/official-image/Dockerfile
> >
> > We noticed that our varnishadm commands started failing. More
> specifically:
> >
> > root at dc17c642d39a:/etc/varnish# varnishadm "ban req.url ~ /"
> > 200
> >
> > As I understand it's a 200 exit code, which means varnishadm failed:
> > https://varnish-cache.org/docs/5.1/reference/varnishadm.html#exit-status
> >
> > What does 200 mean exactly? I couldn't find any code list.
> > My guess is that this has to do with the unprivileged varnish user,
> > but I'm not sure what it takes to fix it.
> >
> >
> > Martynas
> _______________________________________________
> varnish-misc mailing list
> varnish-misc at varnish-cache.org
> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/varnish-misc/attachments/20211002/f4877842/attachment.html>

From martynas at atomgraph.com  Sat Oct  2 22:54:16 2021
From: martynas at atomgraph.com (=?UTF-8?Q?Martynas_Jusevi=C4=8Dius?=)
Date: Sun, 3 Oct 2021 00:54:16 +0200
Subject: varnishadm exit code 200
In-Reply-To: <CAOrPormKS=R0FC94ScLHt23zF5n8CFH9KiHiaqbUNFWS=0zA6g@mail.gmail.com>
References: <CAE35VmxO+eO_4jLyxEWu-pnDZX-9Gw_DfEh6X=n_00Wni068Gg@mail.gmail.com>
 <CAE35Vmw5gSCSEwvSPDcZ_HgoAYu9WJUWgozBAkYpFHCnJOAOUg@mail.gmail.com>
 <CAOrPormKS=R0FC94ScLHt23zF5n8CFH9KiHiaqbUNFWS=0zA6g@mail.gmail.com>
Message-ID: <CAE35VmyXVmpeN0rM14qtaCawVyJpJWDbHE0xojzNULj1roF-fA@mail.gmail.com>

I thought that was a non-zero exit code but it's not :)

We just noticed a change -- in the earlier version we used (not sure
which now, I think 4.x) there was no output from varnishadm.

We'll just send the 200 to /dev/null then.

On Sun, Oct 3, 2021 at 12:39 AM Guillaume Quintard
<guillaume.quintard at gmail.com> wrote:
>
> 200 means the command passed to varnish, via varnishadm, succeeded. What makes you think it failed?
>
> On Sat, Oct 2, 2021, 15:24 Martynas Jusevi?ius <martynas at atomgraph.com> wrote:
>>
>> Actually it does not seem to be the exit code. I tried checking and it
>> looks like the exit code is 0:
>>
>> root at dc17c642d39a:/etc/varnish# varnishadm "ban req.url ~ /"
>> 200
>>
>> root at dc17c642d39a:/etc/varnish# test $? -eq 0 || echo "Error"
>> root at dc17c642d39a:/etc/varnish#
>>
>> So where is that "200" coming from?
>>
>> On Sun, Oct 3, 2021 at 12:14 AM Martynas Jusevi?ius
>> <martynas at atomgraph.com> wrote:
>> >
>> > Hi,
>> >
>> > We recently switched to the varnish:latest container and based an
>> > unprivileged image on it (entrypoint runs as USER varnish):
>> > https://github.com/AtomGraph/varnish/blob/official-image/Dockerfile
>> >
>> > We noticed that our varnishadm commands started failing. More specifically:
>> >
>> > root at dc17c642d39a:/etc/varnish# varnishadm "ban req.url ~ /"
>> > 200
>> >
>> > As I understand it's a 200 exit code, which means varnishadm failed:
>> > https://varnish-cache.org/docs/5.1/reference/varnishadm.html#exit-status
>> >
>> > What does 200 mean exactly? I couldn't find any code list.
>> > My guess is that this has to do with the unprivileged varnish user,
>> > but I'm not sure what it takes to fix it.
>> >
>> >
>> > Martynas
>> _______________________________________________
>> varnish-misc mailing list
>> varnish-misc at varnish-cache.org
>> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc

From guillaume.quintard at gmail.com  Sat Oct  2 23:15:01 2021
From: guillaume.quintard at gmail.com (Guillaume Quintard)
Date: Sat, 2 Oct 2021 16:15:01 -0700
Subject: varnishadm exit code 200
In-Reply-To: <CAE35VmyXVmpeN0rM14qtaCawVyJpJWDbHE0xojzNULj1roF-fA@mail.gmail.com>
References: <CAE35VmxO+eO_4jLyxEWu-pnDZX-9Gw_DfEh6X=n_00Wni068Gg@mail.gmail.com>
 <CAE35Vmw5gSCSEwvSPDcZ_HgoAYu9WJUWgozBAkYpFHCnJOAOUg@mail.gmail.com>
 <CAOrPormKS=R0FC94ScLHt23zF5n8CFH9KiHiaqbUNFWS=0zA6g@mail.gmail.com>
 <CAE35VmyXVmpeN0rM14qtaCawVyJpJWDbHE0xojzNULj1roF-fA@mail.gmail.com>
Message-ID: <CAOrPorkn7wLYZFtiStd0rCcJBAH+BmEFP8asQN=Hfzd-SEOxeA@mail.gmail.com>

Oooooh, 4.x is ancient, glad you made the upgrade!

On Sat, Oct 2, 2021, 15:54 Martynas Jusevi?ius <martynas at atomgraph.com>
wrote:

> I thought that was a non-zero exit code but it's not :)
>
> We just noticed a change -- in the earlier version we used (not sure
> which now, I think 4.x) there was no output from varnishadm.
>
> We'll just send the 200 to /dev/null then.
>
> On Sun, Oct 3, 2021 at 12:39 AM Guillaume Quintard
> <guillaume.quintard at gmail.com> wrote:
> >
> > 200 means the command passed to varnish, via varnishadm, succeeded. What
> makes you think it failed?
> >
> > On Sat, Oct 2, 2021, 15:24 Martynas Jusevi?ius <martynas at atomgraph.com>
> wrote:
> >>
> >> Actually it does not seem to be the exit code. I tried checking and it
> >> looks like the exit code is 0:
> >>
> >> root at dc17c642d39a:/etc/varnish# varnishadm "ban req.url ~ /"
> >> 200
> >>
> >> root at dc17c642d39a:/etc/varnish# test $? -eq 0 || echo "Error"
> >> root at dc17c642d39a:/etc/varnish#
> >>
> >> So where is that "200" coming from?
> >>
> >> On Sun, Oct 3, 2021 at 12:14 AM Martynas Jusevi?ius
> >> <martynas at atomgraph.com> wrote:
> >> >
> >> > Hi,
> >> >
> >> > We recently switched to the varnish:latest container and based an
> >> > unprivileged image on it (entrypoint runs as USER varnish):
> >> > https://github.com/AtomGraph/varnish/blob/official-image/Dockerfile
> >> >
> >> > We noticed that our varnishadm commands started failing. More
> specifically:
> >> >
> >> > root at dc17c642d39a:/etc/varnish# varnishadm "ban req.url ~ /"
> >> > 200
> >> >
> >> > As I understand it's a 200 exit code, which means varnishadm failed:
> >> >
> https://varnish-cache.org/docs/5.1/reference/varnishadm.html#exit-status
> >> >
> >> > What does 200 mean exactly? I couldn't find any code list.
> >> > My guess is that this has to do with the unprivileged varnish user,
> >> > but I'm not sure what it takes to fix it.
> >> >
> >> >
> >> > Martynas
> >> _______________________________________________
> >> varnish-misc mailing list
> >> varnish-misc at varnish-cache.org
> >> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/varnish-misc/attachments/20211002/9c64f345/attachment-0001.html>

From leo.lelievre at protonmail.com  Sun Oct  3 16:05:33 2021
From: leo.lelievre at protonmail.com (=?utf-8?B?TMOpbw==?=)
Date: Sun, 03 Oct 2021 16:05:33 +0000
Subject: Varnish only resolve the ip on startup
Message-ID: <5QFbmzVSZFr8Wa5TKmGoJPKlh_4tM5c5G1y8uaZJou9C2LWim5gJUdWz_eZX2xGiFbc7YQiwKS4viUeaRsABcgMYqwneIda5ynM-22evz-0=@protonmail.com>

Hello,
While setting up a docker-compose with a cache powered by varnish I came across an issue when recreating a container that varnish depends on. After the recreation varnish doesn't try to retrieve the new local ip. I've been told this is an upstream varnish issue (https://github.com/varnish/docker-varnish/issues/41). I have found some ways to solve the problem, but will an appropriate solution be considered ?

Thanks in advance,
Have a good day,
L?o Lelievre
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/varnish-misc/attachments/20211003/b99a64d4/attachment.html>

From dridi at varni.sh  Mon Oct  4 03:52:20 2021
From: dridi at varni.sh (Dridi Boukelmoune)
Date: Mon, 4 Oct 2021 03:52:20 +0000
Subject: varnishadm exit code 200
In-Reply-To: <CAE35Vmw5gSCSEwvSPDcZ_HgoAYu9WJUWgozBAkYpFHCnJOAOUg@mail.gmail.com>
References: <CAE35VmxO+eO_4jLyxEWu-pnDZX-9Gw_DfEh6X=n_00Wni068Gg@mail.gmail.com>
 <CAE35Vmw5gSCSEwvSPDcZ_HgoAYu9WJUWgozBAkYpFHCnJOAOUg@mail.gmail.com>
Message-ID: <CABoVN9Dr1J-d6GYsvGpeOheFyAD=Su2-47Y=rpavP7JG8gDfqA@mail.gmail.com>

On Sat, Oct 2, 2021 at 10:24 PM Martynas Jusevi?ius
<martynas at atomgraph.com> wrote:
>
> Actually it does not seem to be the exit code. I tried checking and it
> looks like the exit code is 0:
>
> root at dc17c642d39a:/etc/varnish# varnishadm "ban req.url ~ /"
> 200
>
> root at dc17c642d39a:/etc/varnish# test $? -eq 0 || echo "Error"
> root at dc17c642d39a:/etc/varnish#
>
> So where is that "200" coming from?

You shouldn't see a 200, except for this bug fixed in 7.0:

https://github.com/varnishcache/varnish-cache/issues/3687

What version of Varnish are you running?

Dridi

From guillaume.quintard at gmail.com  Mon Oct  4 15:43:19 2021
From: guillaume.quintard at gmail.com (Guillaume Quintard)
Date: Mon, 4 Oct 2021 08:43:19 -0700
Subject: Varnish only resolve the ip on startup
In-Reply-To: <5QFbmzVSZFr8Wa5TKmGoJPKlh_4tM5c5G1y8uaZJou9C2LWim5gJUdWz_eZX2xGiFbc7YQiwKS4viUeaRsABcgMYqwneIda5ynM-22evz-0=@protonmail.com>
References: <5QFbmzVSZFr8Wa5TKmGoJPKlh_4tM5c5G1y8uaZJou9C2LWim5gJUdWz_eZX2xGiFbc7YQiwKS4viUeaRsABcgMYqwneIda5ynM-22evz-0=@protonmail.com>
Message-ID: <CAOrPorkLT+SvnrruoNLO21+U4_FeA=rZ_k3SZRpJySSsT-G7fA@mail.gmail.com>

I think it makes sense for Varnish to natively support backends changing
their IPs. I do get the performance argument but now that there is a
cloud/container market and that Varnish has proven to be useful in it, this
basic functionality should be brought in.

Would it be acceptable to add a "host_string" to vrt_endpoint and fill it
if the VCL backend isn't an IP, then, we can add another cp_methods to
cache_conn_pool.c to use it? This way IPs are still super fast, and
hostnames become actually useful and a bit less confusing?

-- 
Guillaume Quintard


On Sun, Oct 3, 2021 at 9:07 AM L?o <leo.lelievre at protonmail.com> wrote:

> Hello,
> While setting up a docker-compose with a cache powered by varnish I came
> across an issue when recreating a container that varnish depends on. After
> the recreation varnish doesn't try to retrieve the new local ip. I've been
> told this is an upstream varnish issue (
> https://github.com/varnish/docker-varnish/issues/41). I have found some
> ways to solve the problem, but will an appropriate solution be considered ?
>
> Thanks in advance,
> Have a good day,
> L?o Lelievre
> _______________________________________________
> varnish-misc mailing list
> varnish-misc at varnish-cache.org
> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/varnish-misc/attachments/20211004/f3839fff/attachment.html>

From dridi at varni.sh  Mon Oct  4 16:48:36 2021
From: dridi at varni.sh (Dridi Boukelmoune)
Date: Mon, 4 Oct 2021 16:48:36 +0000
Subject: Varnish only resolve the ip on startup
In-Reply-To: <CAOrPorkLT+SvnrruoNLO21+U4_FeA=rZ_k3SZRpJySSsT-G7fA@mail.gmail.com>
References: <5QFbmzVSZFr8Wa5TKmGoJPKlh_4tM5c5G1y8uaZJou9C2LWim5gJUdWz_eZX2xGiFbc7YQiwKS4viUeaRsABcgMYqwneIda5ynM-22evz-0=@protonmail.com>
 <CAOrPorkLT+SvnrruoNLO21+U4_FeA=rZ_k3SZRpJySSsT-G7fA@mail.gmail.com>
Message-ID: <CABoVN9BUYmPGcaqh6rHV==Oik2b4AedRjxYwBoh8F84+yWfU9w@mail.gmail.com>

On Mon, Oct 4, 2021 at 3:45 PM Guillaume Quintard
<guillaume.quintard at gmail.com> wrote:
>
> I think it makes sense for Varnish to natively support backends changing their IPs. I do get the performance argument but now that there is a cloud/container market and that Varnish has proven to be useful in it, this basic functionality should be brought in.

I would assume the primary argument was simplicity, not performance,
but I wasn't around. One could argue it's turned into simplism in
today's cloudy cloud cloudy world.

> Would it be acceptable to add a "host_string" to vrt_endpoint and fill it if the VCL backend isn't an IP, then, we can add another cp_methods to cache_conn_pool.c to use it? This way IPs are still super fast, and hostnames become actually useful and a bit less confusing?

One problem I have (and that you should be familiar with) is that
portable interfaces we have that *respect* the system configuration
(hosts file, nsswitch configuration etc) are not providing enough
information. For example it becomes cumbersome to resolve SRV records
or even get the TTL of individual records for a DNS resolution in a
*portable* fashion.

When you put it like this, it sounds simple enough (dare I say
simplistic?) but what I see is a sizeable can of worms.

I do think we could do something about it, I don't know what would be
satisfying.

Cheers,
Dridi

From guillaume.quintard at gmail.com  Mon Oct  4 17:15:47 2021
From: guillaume.quintard at gmail.com (Guillaume Quintard)
Date: Mon, 4 Oct 2021 10:15:47 -0700
Subject: Varnish only resolve the ip on startup
In-Reply-To: <CABoVN9BUYmPGcaqh6rHV==Oik2b4AedRjxYwBoh8F84+yWfU9w@mail.gmail.com>
References: <5QFbmzVSZFr8Wa5TKmGoJPKlh_4tM5c5G1y8uaZJou9C2LWim5gJUdWz_eZX2xGiFbc7YQiwKS4viUeaRsABcgMYqwneIda5ynM-22evz-0=@protonmail.com>
 <CAOrPorkLT+SvnrruoNLO21+U4_FeA=rZ_k3SZRpJySSsT-G7fA@mail.gmail.com>
 <CABoVN9BUYmPGcaqh6rHV==Oik2b4AedRjxYwBoh8F84+yWfU9w@mail.gmail.com>
Message-ID: <CAOrPor=7ceaPqmE=7RYxiWfAhLW4vyc9VqDHQ_Mv4vK_B+4LVA@mail.gmail.com>

On Mon, Oct 4, 2021 at 9:49 AM Dridi Boukelmoune <dridi at varni.sh> wrote:

> One problem I have (and that you should be familiar with) is that
> portable interfaces we have that *respect* the system configuration
> (hosts file, nsswitch configuration etc) are not providing enough
> information. For example it becomes cumbersome to resolve SRV records
> or even get the TTL of individual records for a DNS resolution in a
> *portable* fashion.
>
> When you put it like this, it sounds simple enough (dare I say
> simplistic?) but what I see is a sizeable can of worms.
>

That sounds like a bit of a strawman to me. getaddrinfo and connect are
standard, and that's about all we should need. Applications are supposed
(in general) to just use whatever gai give them. We can call them every
time we need a new connection so we don't worry about TTL, and we just
disregard SRV records.
The vast majority of users don't need SRV (yet?), and don't expect the
application to optimize DNS calls, but they do complain that giving a
hostname to VCL doesn't work.

Let's just provide basic, expected functionality out of the box, and leave
the fancier features to vmod_goto and vmod_dynamic.

--
Guillaume Quintard
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/varnish-misc/attachments/20211004/3ce9c519/attachment-0001.html>

From justinl at arena.net  Mon Oct  4 17:29:55 2021
From: justinl at arena.net (Justin Lloyd)
Date: Mon, 4 Oct 2021 17:29:55 +0000
Subject: Varnish only resolve the ip on startup
In-Reply-To: <CAOrPor=7ceaPqmE=7RYxiWfAhLW4vyc9VqDHQ_Mv4vK_B+4LVA@mail.gmail.com>
References: <5QFbmzVSZFr8Wa5TKmGoJPKlh_4tM5c5G1y8uaZJou9C2LWim5gJUdWz_eZX2xGiFbc7YQiwKS4viUeaRsABcgMYqwneIda5ynM-22evz-0=@protonmail.com>
 <CAOrPorkLT+SvnrruoNLO21+U4_FeA=rZ_k3SZRpJySSsT-G7fA@mail.gmail.com>
 <CABoVN9BUYmPGcaqh6rHV==Oik2b4AedRjxYwBoh8F84+yWfU9w@mail.gmail.com>
 <CAOrPor=7ceaPqmE=7RYxiWfAhLW4vyc9VqDHQ_Mv4vK_B+4LVA@mail.gmail.com>
Message-ID: <MWHPR1001MB2174D1498876582B9657BBD5A1AE9@MWHPR1001MB2174.namprd10.prod.outlook.com>

I?m definitely watching this topic, considering I?m planning on moving to Varnish Enterprise next year and putting a cluster in ECS, if not Fargate, so being able to easily handle dynamic IPs would be extremely helpful.

From: varnish-misc <varnish-misc-bounces+justinl=arena.net at varnish-cache.org> On Behalf Of Guillaume Quintard
Sent: Monday, October 4, 2021 10:16 AM
To: Dridi Boukelmoune <dridi at varni.sh>
Cc: varnish-misc at varnish-cache.org
Subject: Re: Varnish only resolve the ip on startup

On Mon, Oct 4, 2021 at 9:49 AM Dridi Boukelmoune <dridi at varni.sh<mailto:dridi at varni.sh>> wrote:
One problem I have (and that you should be familiar with) is that
portable interfaces we have that *respect* the system configuration
(hosts file, nsswitch configuration etc) are not providing enough
information. For example it becomes cumbersome to resolve SRV records
or even get the TTL of individual records for a DNS resolution in a
*portable* fashion.

When you put it like this, it sounds simple enough (dare I say
simplistic?) but what I see is a sizeable can of worms.

That sounds like a bit of a strawman to me. getaddrinfo and connect are standard, and that's about all we should need. Applications are supposed (in general) to just use whatever gai give them. We can call them every time we need a new connection so we don't worry about TTL, and we just disregard SRV records.
The vast majority of users don't need SRV (yet?), and don't expect the application to optimize DNS calls, but they do complain that giving a hostname to VCL doesn't work.

Let's just provide basic, expected functionality out of the box, and leave the fancier features to vmod_goto and vmod_dynamic.

--
Guillaume Quintard
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/varnish-misc/attachments/20211004/068adcb2/attachment.html>

From dridi at varni.sh  Mon Oct  4 18:38:11 2021
From: dridi at varni.sh (Dridi Boukelmoune)
Date: Mon, 4 Oct 2021 18:38:11 +0000
Subject: Varnish only resolve the ip on startup
In-Reply-To: <MWHPR1001MB2174D1498876582B9657BBD5A1AE9@MWHPR1001MB2174.namprd10.prod.outlook.com>
References: <5QFbmzVSZFr8Wa5TKmGoJPKlh_4tM5c5G1y8uaZJou9C2LWim5gJUdWz_eZX2xGiFbc7YQiwKS4viUeaRsABcgMYqwneIda5ynM-22evz-0=@protonmail.com>
 <CAOrPorkLT+SvnrruoNLO21+U4_FeA=rZ_k3SZRpJySSsT-G7fA@mail.gmail.com>
 <CABoVN9BUYmPGcaqh6rHV==Oik2b4AedRjxYwBoh8F84+yWfU9w@mail.gmail.com>
 <CAOrPor=7ceaPqmE=7RYxiWfAhLW4vyc9VqDHQ_Mv4vK_B+4LVA@mail.gmail.com>
 <MWHPR1001MB2174D1498876582B9657BBD5A1AE9@MWHPR1001MB2174.namprd10.prod.outlook.com>
Message-ID: <CABoVN9CUEYRYee914Or6FJ-3gVtCoKLgPh42dVS5YpWU8S55JQ@mail.gmail.com>

On Mon, Oct 4, 2021 at 5:29 PM Justin Lloyd <justinl at arena.net> wrote:
>
> I?m definitely watching this topic, considering I?m planning on moving to Varnish Enterprise next year and putting a cluster in ECS, if not Fargate, so being able to easily handle dynamic IPs would be extremely helpful.

Like I implied in my initial response you have this ability out of the
box with Varnish Enterprise, and with Varnish Cache there are
third-party VMODs.

Dridi

From dridi at varni.sh  Mon Oct  4 18:54:56 2021
From: dridi at varni.sh (Dridi Boukelmoune)
Date: Mon, 4 Oct 2021 18:54:56 +0000
Subject: Varnish only resolve the ip on startup
In-Reply-To: <CAOrPor=7ceaPqmE=7RYxiWfAhLW4vyc9VqDHQ_Mv4vK_B+4LVA@mail.gmail.com>
References: <5QFbmzVSZFr8Wa5TKmGoJPKlh_4tM5c5G1y8uaZJou9C2LWim5gJUdWz_eZX2xGiFbc7YQiwKS4viUeaRsABcgMYqwneIda5ynM-22evz-0=@protonmail.com>
 <CAOrPorkLT+SvnrruoNLO21+U4_FeA=rZ_k3SZRpJySSsT-G7fA@mail.gmail.com>
 <CABoVN9BUYmPGcaqh6rHV==Oik2b4AedRjxYwBoh8F84+yWfU9w@mail.gmail.com>
 <CAOrPor=7ceaPqmE=7RYxiWfAhLW4vyc9VqDHQ_Mv4vK_B+4LVA@mail.gmail.com>
Message-ID: <CABoVN9BEJtK=SNXFKxQrx_t9T+FipwybgkrBSMao7-Z4Jm72Xw@mail.gmail.com>

On Mon, Oct 4, 2021 at 5:15 PM Guillaume Quintard
<guillaume.quintard at gmail.com> wrote:
>
> On Mon, Oct 4, 2021 at 9:49 AM Dridi Boukelmoune <dridi at varni.sh> wrote:
>>
>> One problem I have (and that you should be familiar with) is that
>> portable interfaces we have that *respect* the system configuration
>> (hosts file, nsswitch configuration etc) are not providing enough
>> information. For example it becomes cumbersome to resolve SRV records
>> or even get the TTL of individual records for a DNS resolution in a
>> *portable* fashion.
>>
>> When you put it like this, it sounds simple enough (dare I say
>> simplistic?) but what I see is a sizeable can of worms.
>
>
> That sounds like a bit of a strawman to me. getaddrinfo and connect are standard, and that's about all we should need. Applications are supposed (in general) to just use whatever gai give them. We can call them every time we need a new connection so we don't worry about TTL, and we just disregard SRV records.
> The vast majority of users don't need SRV (yet?), and don't expect the application to optimize DNS calls, but they do complain that giving a hostname to VCL doesn't work.

The "portable" interface I was referring to includes getaddrinfo, and
I guess your suggestion would be to always resolve and leave A/AAAA
records caching to your stub and/or recursive resolver.

Fair enough, simplicity.

That still doesn't seal the can of worms: once there are more than one
address per family or addresses change, it's our connection and
pooling models that need to be revisited, how many different addresses
to try connecting to, how to retry, and how we account for all of that
(stats for example). Again, it's a bit more complex that just saying
"change the connect callback to one that combines resolve+connect".

I'm not against a holistic management of connections, there are
actually more aspects that I think should be at the core of our
connection management like certificates attached to a session (and in
particular their domains) once we have TLS support.

I just think it needs to be better defined and not give the impression
that messing about a new set of backend callbacks that will
systematically resolve endpoints is as simple as it gets.

Cheers

From guillaume.quintard at gmail.com  Tue Oct  5 03:11:28 2021
From: guillaume.quintard at gmail.com (Guillaume Quintard)
Date: Mon, 4 Oct 2021 20:11:28 -0700
Subject: Varnish only resolve the ip on startup
In-Reply-To: <CABoVN9BEJtK=SNXFKxQrx_t9T+FipwybgkrBSMao7-Z4Jm72Xw@mail.gmail.com>
References: <5QFbmzVSZFr8Wa5TKmGoJPKlh_4tM5c5G1y8uaZJou9C2LWim5gJUdWz_eZX2xGiFbc7YQiwKS4viUeaRsABcgMYqwneIda5ynM-22evz-0=@protonmail.com>
 <CAOrPorkLT+SvnrruoNLO21+U4_FeA=rZ_k3SZRpJySSsT-G7fA@mail.gmail.com>
 <CABoVN9BUYmPGcaqh6rHV==Oik2b4AedRjxYwBoh8F84+yWfU9w@mail.gmail.com>
 <CAOrPor=7ceaPqmE=7RYxiWfAhLW4vyc9VqDHQ_Mv4vK_B+4LVA@mail.gmail.com>
 <CABoVN9BEJtK=SNXFKxQrx_t9T+FipwybgkrBSMao7-Z4Jm72Xw@mail.gmail.com>
Message-ID: <CAOrPorkzAZKiMC+g6OHMAvRFF+sg3Q6rxR2yakvii3gucHB0CA@mail.gmail.com>

On Mon, Oct 4, 2021 at 11:55 AM Dridi Boukelmoune <dridi at varni.sh> wrote:

> That still doesn't seal the can of worms: once there are more than one
> address per family or addresses change, it's our connection and
> pooling models that need to be revisited, how many different addresses
> to try connecting to, how to retry, and how we account for all of that
> (stats for example). Again, it's a bit more complex that just saying
> "change the connect callback to one that combines resolve+connect".
>

I do understand that it's the core of the problem, and I'm probably being
pig-headed on this, but it feels to me that it's not really different from
a server with a floating IP, or a level-4 load-balancer fronting the
backend. The addresses may change, there may be more than one, but once the
connection is open, you can trust it and keep using it.
We trust connect() to get us to our goal, using getaddrinfo() on top of it
just means that we trust the DNS server to provide good info. So:
- if we prefer IPv6, we go through the list and pick the first IPv6 entry,
if we don't find one, we just grab the first IPv4 entry, no second chance
- if you try to reuse a connection and it died on you, you try to
resolve+connect a new one. Maybe you get the same IP, maybe you don't, but
we trust the DNS implementation to shuffle the entries around
- stats is an interesting issue, but again, only if you let it be. Backends
are already "IP address" or "UDS path", I'm fine with hiding all the DNS
entries behind "non-numerical host".

I agree, things get gnarly when we start fiddling with DNs, but there's
that portable interface that allows us to not have to. And if you want
fancy stuff like stats per IP, and fun load-balancing options that are
DNS-aware, there are vmods for that.

For the sake of transparency, I must admit that I do have an issue with my
proposal though: there's no timeout option for getaddrinfo, which sucks.

-- 
Guillaume Quintard
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/varnish-misc/attachments/20211004/4632e71b/attachment.html>

From slink at schokola.de  Tue Oct  5 13:38:49 2021
From: slink at schokola.de (Nils Goroll)
Date: Tue, 5 Oct 2021 15:38:49 +0200
Subject: Varnish only resolve the ip on startup
In-Reply-To: <5QFbmzVSZFr8Wa5TKmGoJPKlh_4tM5c5G1y8uaZJou9C2LWim5gJUdWz_eZX2xGiFbc7YQiwKS4viUeaRsABcgMYqwneIda5ynM-22evz-0=@protonmail.com>
References: <5QFbmzVSZFr8Wa5TKmGoJPKlh_4tM5c5G1y8uaZJou9C2LWim5gJUdWz_eZX2xGiFbc7YQiwKS4viUeaRsABcgMYqwneIda5ynM-22evz-0=@protonmail.com>
Message-ID: <059d0011-41a9-a82c-bd8c-1ba467c85d53@schokola.de>

On 03.10.21 18:05, L?o wrote:
> I have found some ways to solve the problem

https://github.com/nigoroll/libvmod-dynamic exists to do just that


From guillaume.quintard at gmail.com  Tue Oct  5 17:25:24 2021
From: guillaume.quintard at gmail.com (Guillaume Quintard)
Date: Tue, 5 Oct 2021 10:25:24 -0700
Subject: Varnish only resolve the ip on startup
In-Reply-To: <059d0011-41a9-a82c-bd8c-1ba467c85d53@schokola.de>
References: <5QFbmzVSZFr8Wa5TKmGoJPKlh_4tM5c5G1y8uaZJou9C2LWim5gJUdWz_eZX2xGiFbc7YQiwKS4viUeaRsABcgMYqwneIda5ynM-22evz-0=@protonmail.com>
 <059d0011-41a9-a82c-bd8c-1ba467c85d53@schokola.de>
Message-ID: <CAOrPor=Ctkf=dZVBYinqt2_oMOY+SCpDBZZSCO1A8xxjxTjxzQ@mail.gmail.com>

I'm fine with adding vmod_dynamic to the varnish docker images so that
users have access to this basic functionality, but, for the record:
- I really feel this should be considered a basic feature and exist in core
- if the docker image starts adding downstream vmods, it opens the gates to
a flood of "can you add this vmod too?" questions, which I'm not looking
forward to.

-- 
Guillaume Quintard


On Tue, Oct 5, 2021 at 6:39 AM Nils Goroll <slink at schokola.de> wrote:

> On 03.10.21 18:05, L?o wrote:
> > I have found some ways to solve the problem
>
> https://github.com/nigoroll/libvmod-dynamic exists to do just that
>
> _______________________________________________
> varnish-misc mailing list
> varnish-misc at varnish-cache.org
> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/varnish-misc/attachments/20211005/54fb725b/attachment.html>

From leo.lelievre at protonmail.com  Tue Oct  5 17:47:18 2021
From: leo.lelievre at protonmail.com (=?utf-8?B?TMOpbw==?=)
Date: Tue, 05 Oct 2021 17:47:18 +0000
Subject: Varnish only resolve the ip on startup
In-Reply-To: <059d0011-41a9-a82c-bd8c-1ba467c85d53@schokola.de>
References: <5QFbmzVSZFr8Wa5TKmGoJPKlh_4tM5c5G1y8uaZJou9C2LWim5gJUdWz_eZX2xGiFbc7YQiwKS4viUeaRsABcgMYqwneIda5ynM-22evz-0=@protonmail.com>
 <059d0011-41a9-a82c-bd8c-1ba467c85d53@schokola.de>
Message-ID: <y-ALCIYppd-yrPHp7I9SSthxR-knFf9Sd3uufzPbiEgtZe2dukP5QQxmndJ0UFxjXLK130OTh2J5vNyn7U0qG0HlGroaFwgcmPdAAFTwIA8=@protonmail.com>

mailto:slink at schokola.de

> On 03.10.21 18:05, L?o wrote:
>> I have found some ways to solve the problem
>
> https://github.com/nigoroll/libvmod-dynamic exists to do just that

The point was to keep the docker compose as simple as possible without the need to install other services
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/varnish-misc/attachments/20211005/edf1cdd9/attachment.html>

From dridi at varni.sh  Wed Oct  6 05:46:40 2021
From: dridi at varni.sh (Dridi Boukelmoune)
Date: Wed, 6 Oct 2021 05:46:40 +0000
Subject: Difference between adjusting set req backend in vcl_recv and
 vcl_backend_fetch
In-Reply-To: <MW2PR16MB2361E7753DAD78BD8646077AD9D29@MW2PR16MB2361.namprd16.prod.outlook.com>
References: <MW2PR16MB2361E7753DAD78BD8646077AD9D29@MW2PR16MB2361.namprd16.prod.outlook.com>
Message-ID: <CABoVN9CO981YTzGBU9i8PRfQ=JHkDBQKDjOKUGdX0V8xGmFpig@mail.gmail.com>

On Mon, Sep 6, 2021 at 2:22 PM Hamidreza Hosseini
<hrhosseini at hotmail.com> wrote:
>
> Hi,
> This is part of my varnish configuration file for sharding:
>
> ```
> cat /etc/varnish/default.vcl
>
> vcl 4.1;
>
> import directors;
>
>
> probe myprobe {
>     .request =
>       "HEAD /healthcheck.php HTTP/1.1"
>       "Connection: close"
>       "User-Agent: Varnish Health Probe";
>     .timeout = 1s;
>     .interval = 5s;
>     .window = 5;
>     .threshold = 3;
> }
>
>
> backend b-01 { .host = "b-01"; .port = "80"; .probe = myprobe; }
> backend b-02 { .host = "b-02"; .port = "80"; .probe = myprobe; }
> backend b-03 { .host = "b-03"; .port = "80"; .probe = myprobe; }
>
>
> sub vcl_init {
>
>
>   new hls_cluster = directors.shard();
>     hls_cluster.add_backend(b-01);
>     hls_cluster.add_backend(b-02);
>     hls_cluster.add_backend(b-03);
>
>
>   new p = directors.shard_param();
>
>     hls_cluster.set_rampup(30s);
> #    hls_cluster.set_warmup(0.5);
>
>     hls_cluster.reconfigure();
>     hls_cluster.associate(p.use());
>
> }
>
>
>
> acl purge {
>     "localhost";
> }
>
>
> sub vcl_recv {
>
>    set req.backend_hint = hls_cluster.backend();
>
> }
>
>
> sub vcl_backend_fetch {
>
>   p.set(by=KEY, key=hls_cluster.key(bereq.url));
>   set bereq.backend = hls_cluster.backend(resolve=LAZY, healthy=CHOSEN);
>
> }
>
> ```
> 1. there are two set backend in this config, one is on vcl_recv:
> "set req.backend_hint = hls_cluster.backend();"
> and one in vcl_backend_fetch:
> "set bereq.backend = hls_cluster.backend(resolve=LAZY, healthy=CHOSEN);"
> should I remove set in vcl_recv cause I think if I adjust it , all requset will go through this backend list and configs like healthy=CHOSEN in vcl_backend_fetch wouldnt be applied! Am I true?

bereq.backend is initialized from req.backend_hint when a backend
fetch is triggered. Setting bereq.backend will simply override
anything one in vcl_recv.

> 2.Actually what is difference between vcl_backend_fetch and vcl_recv?

In vcl_recv you are processing a client request that was just received.

In vcl_backend_fetch you are preparing a bereq (backend request)
derived from req (client request) in before fetching the resource from
the backend.

https://varnish-cache.org/docs/6.0/users-guide/vcl-built-in-subs.html

> 3.should I remove "set req.backend_hint" from vcl_recv?

If the answer to "do I use the backend selection to make decisions in
client subroutines?" is no, then you can remove it from vcl_recv.

The shard director has task-specific settings, so some things you may
configure in vcl_recv would not apply to vcl_backend_fetch, so if the
answer to the question above was yes, you would probably need both.

Best,
Dridi

From dridi at varni.sh  Wed Oct  6 06:12:09 2021
From: dridi at varni.sh (Dridi Boukelmoune)
Date: Wed, 6 Oct 2021 06:12:09 +0000
Subject: Best practice for caching scenario with different backend servers
 but same content
In-Reply-To: <MW2PR16MB236129422381FF9D95DF8A69D9FD9@MW2PR16MB2361.namprd16.prod.outlook.com>
References: <MW2PR16MB23610AE0B323CD64A612E30ED9EE9@MW2PR16MB2361.namprd16.prod.outlook.com>
 <MW2PR16MB2361E21D1A5C75993DBE5F5FD9FB9@MW2PR16MB2361.namprd16.prod.outlook.com>
 <CABoVN9AN1d=stdU6s_JevVaP3Nw8kG+urVqQnqrHSWUZKJdqRQ@mail.gmail.com>
 <MW2PR16MB236129422381FF9D95DF8A69D9FD9@MW2PR16MB2361.namprd16.prod.outlook.com>
Message-ID: <CABoVN9BD12NEX51jSZSa2wsxUuw5AOdjaLFeQx5AEOB1LFw-3Q@mail.gmail.com>

On Mon, Aug 16, 2021 at 1:34 PM Hamidreza Hosseini
<hrhosseini at hotmail.com> wrote:
>
> > In that case, hashing the URL only would prevent you from adding new
> domains through your Varnish server. It won't hurt if you know you
> will only ever have one domain to deal with, but hashing the host will
> also not hurt as long as you normalize it to a unique value.
>
> Hi,
> Let me elaborate my architecture more:
> I have some backend servers to serve hls fragments for video live stream,e.g:
>
> ```
>
> hls_backend_01
> hls_backend_02
> hls_backend_03
> hls_backend_04
> hls_backend_05
> hls_backend_06
> hls_backend_07
> hls_backend_08
> hls_backend_09
> hls_backend_10
>
> ```
>
> There is same content on all hls backend servers, there are 5 varnish in front of them for caching
> Now If I use round-robin director on Varnishes, because varnish would cache " req.http.host + req.url ", so for the same content but from different backends it would cache double! for example:
> if varnish for the first request and "test.ts" file goes to "hls_backend_01"  backend server, would cache it and
> for the next request from other clients because it is using round-robin director
> it goes to "hls_backend_02" and would cache the same file again due to different "req.http.host"
> So now I have a solution to use Shard director based on "key=req.url" instead of round robin
> another way is to use round robin but adjusting the hash vcl to something like bellow:
>
> ```
>
> sub vcl_hash {
>     hash_data(req.url);
>     return (lookup);
> }
>
> ```
>
> In this way varnish just hash the "req.url" not "req.http.host"
> So, Varnish would cache the content based on the content uniqueness not based on the difference between backends.
> 1. At first, I asked how I can normalize it, Is it possible at all according to what I said!?
> Would you please explain it more with an example?

In this case I think you are confusing "req.http.host" (host header)
with the backend host name.

For example, if you reach one of your 5 Varnish servers via
www.example.com that's what clients will use and that's what
req.http.host will contain.

Your backends FQDNs could be something like this:

- hls01.internal.example.com
- hls02.internal.example.com
- hls03.internal.example.com
- ...
- hls10.internal.example.com

As the example suggests, these domains should not be directly reached
by clients if your goal is to proxy them with Varnish. Those internal
FQDNs should have no effect on the cache key populated with
hash_data(...).

> 2. You give an example about other domains, In this case I do not understand what it has to do with the domain

Let's say your clients can reach either example.com or www.example.com
for the same service, or tomorrow you add more than your HLS service
behind Varnish you may very well receive multiple host headers.

> 3.Maybe I'm thinking in wrong way because if varnish hash the data based on req.url : 'hash_data(req.url)' It shouldn't cache the same content but different backends again!
> for example my request is :

In this case you are "hashing" the client request with hash_data(...)
and it has nothing to do with backend selection. The fallback director
will precisely not do any kind of traffic balancing since its purpose
is to always select the first healthy backend in the insertion order.
The shard director may rely on the request hash or other criteria as
we already covered.

> http://varnish-01:/hls/test.ts
> for first request it goes to "hls_backend_01" backend and cache it and for next request it goes to "hls_backend_02" backend,
> so for each request it caches it again because backends are different?

All subsequent requests to http://varnish-01:/hls/test.ts should go to
the same hls_backend_01 backend with the shard director. As long as
there are no other criteria than the ones we already discussed. If you
want consistency across all your Varnish servers, you should configure
your shard directors identically, with the backends added in the same
order (unlike your initial VCL example using the fallback director).

Best,
Dridi

From justinl at arena.net  Mon Oct 25 18:21:54 2021
From: justinl at arena.net (Justin Lloyd)
Date: Mon, 25 Oct 2021 18:21:54 +0000
Subject: Query for authorization username
Message-ID: <MWHPR1001MB21746C46BF20B6F833FB7DD5A1839@MWHPR1001MB2174.namprd10.prod.outlook.com>

In my dev environment, I have a few users configured to use Basic authorization (configured in the Nginx backend) and I'd like to be able to perform VSL queries based on the auth user. This is what I was able to come up with, but I'm wondering if there is a simpler way that I'm just not seeing.

require blob;
if (req.http.Authorization) {
    set req.http.X-Auth-User = regsub(blob.encode(IDENTITY,
                                                  blob=blob.decode(BASE64,
                                                                   encoded=regsub(req.http.Authorization, "^Basic (.*)", "\1"))),
                                      ":.*$", "");
}

varnishtop -I ReqHeader:X-Auth-User
varnishlog -i ReqURL -q 'ReqHeader:X-Auth-User ~ "someuser"'

Thanks,
Justin

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/varnish-misc/attachments/20211025/feb5580b/attachment.html>

From guillaume.quintard at gmail.com  Mon Oct 25 21:58:27 2021
From: guillaume.quintard at gmail.com (Guillaume Quintard)
Date: Mon, 25 Oct 2021 14:58:27 -0700
Subject: Query for authorization username
In-Reply-To: <MWHPR1001MB21746C46BF20B6F833FB7DD5A1839@MWHPR1001MB2174.namprd10.prod.outlook.com>
References: <MWHPR1001MB21746C46BF20B6F833FB7DD5A1839@MWHPR1001MB2174.namprd10.prod.outlook.com>
Message-ID: <CAOrPornAPFPPpa0DpOKJJ+m=FF5zwCBPQeSGkW4eN6DhAdPvRA@mail.gmail.com>

I think it's close to optimal, given the current tools. I would probably
try to move away from regsub() and use vmod_str (
https://github.com/varnish/varnish-modules/blob/master/src/vmod_str.vcc#L42),
and maaaaaaybe use multiple assignments rather than on big expressions, but
that's a personal preference at this point.

It would look like something like this in my mind (highly untested, don't
sue me if your computer explodes):

import var;
import str;
import blob;

sub vcl_recv {
    if (str.split(req.http.Authorization, 0) == "Basic") {
        var.set("b64", str.split(req.http.Authorization, 1));
        var.set("decoded", bob.transcode(encoding = BASE64URL, encoded =
var.get("b64")));
        set req.http.X-Auth-User = str.split(var.get("decoded"), 0, ":");
    }
}


everything in one expression:

set req.http.X-Auth-User = str.split(
                               blob.transcode(
                                   encoding = BASE64URL,
                                   encoded = str.split(req.http.Authorization,
1))
                               ),
                               0,
                               ":"
                           );


You should possibly use blob.transcode() anyway.

--
Guillaume Quintard


On Mon, Oct 25, 2021 at 11:25 AM Justin Lloyd <justinl at arena.net> wrote:

> In my dev environment, I have a few users configured to use Basic
> authorization (configured in the Nginx backend) and I?d like to be able to
> perform VSL queries based on the auth user. This is what I was able to come
> up with, but I?m wondering if there is a simpler way that I?m just not
> seeing.
>
>
>
> require blob;
>
> if (req.http.Authorization) {
>
>     set req.http.X-Auth-User = regsub(blob.encode(IDENTITY,
>
>                                                   blob=blob.decode(BASE64,
>
>
> encoded=regsub(req.http.Authorization, "^Basic (.*)", "\1"))),
>
>                                       ":.*$", "");
>
> }
>
>
>
> varnishtop -I ReqHeader:X-Auth-User
>
> varnishlog -i ReqURL -q 'ReqHeader:X-Auth-User ~ ?someuser?'
>
>
>
> Thanks,
>
> Justin
>
>
> _______________________________________________
> varnish-misc mailing list
> varnish-misc at varnish-cache.org
> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/varnish-misc/attachments/20211025/dae4de00/attachment-0001.html>

From justinl at arena.net  Tue Oct 26 19:01:32 2021
From: justinl at arena.net (Justin Lloyd)
Date: Tue, 26 Oct 2021 19:01:32 +0000
Subject: Query for authorization username
In-Reply-To: <CAOrPornAPFPPpa0DpOKJJ+m=FF5zwCBPQeSGkW4eN6DhAdPvRA@mail.gmail.com>
References: <MWHPR1001MB21746C46BF20B6F833FB7DD5A1839@MWHPR1001MB2174.namprd10.prod.outlook.com>
 <CAOrPornAPFPPpa0DpOKJJ+m=FF5zwCBPQeSGkW4eN6DhAdPvRA@mail.gmail.com>
Message-ID: <MWHPR1001MB2174621F34B54A91D96AEF1DA1849@MWHPR1001MB2174.namprd10.prod.outlook.com>

Hi Guillaume,

Thanks for the feedback! I?m hesitant about adding the external modules just due to the additional complexity of building and maintaining them via my configuration management system (SaltStack) right now.

Here?s what works for me by changing to blob.transcode() while still using regsub():

set req.http.X-Auth-User = regsub(blob.transcode(
                                      encoding=IDENTITY,
                                      decoding=BASE64,
                                      encoded=regsub(req.http.Authorization, "^Basic (.*)", "\1")),
                                  ":.*$",
                                  ""
                           );

Best,
Justin


From: Guillaume Quintard <guillaume.quintard at gmail.com>
Sent: Monday, October 25, 2021 2:58 PM
To: Justin Lloyd <justinl at arena.net>
Cc: varnish-misc at varnish-cache.org
Subject: Re: Query for authorization username

I think it's close to optimal, given the current tools. I would probably try to move away from regsub() and use vmod_str (https://github.com/varnish/varnish-modules/blob/master/src/vmod_str.vcc#L42), and maaaaaaybe use multiple assignments rather than on big expressions, but that's a personal preference at this point.

It would look like something like this in my mind (highly untested, don't sue me if your computer explodes):

import var;
import str;
import blob;

sub vcl_recv {
    if (str.split(req.http.Authorization, 0) == "Basic") {
        var.set("b64", str.split(req.http.Authorization, 1));
        var.set("decoded", bob.transcode(encoding = BASE64URL, encoded = var.get("b64")));
        set req.http.X-Auth-User = str.split(var.get("decoded"), 0, ":");
    }
}

everything in one expression:

set req.http.X-Auth-User = str.split(
                               blob.transcode(
                                   encoding = BASE64URL,
                                   encoded = str.split(req.http.Authorization, 1))
                               ),
                               0,
                               ":"
                           );

You should possibly use blob.transcode() anyway.

--
Guillaume Quintard


On Mon, Oct 25, 2021 at 11:25 AM Justin Lloyd <justinl at arena.net<mailto:justinl at arena.net>> wrote:
In my dev environment, I have a few users configured to use Basic authorization (configured in the Nginx backend) and I?d like to be able to perform VSL queries based on the auth user. This is what I was able to come up with, but I?m wondering if there is a simpler way that I?m just not seeing.

require blob;
if (req.http.Authorization) {
    set req.http.X-Auth-User = regsub(blob.encode(IDENTITY,
                                                  blob=blob.decode(BASE64,
                                                                   encoded=regsub(req.http.Authorization, "^Basic (.*)", "\1"))),
                                      ":.*$", "");
}

varnishtop -I ReqHeader:X-Auth-User
varnishlog -i ReqURL -q 'ReqHeader:X-Auth-User ~ ?someuser?'

Thanks,
Justin

_______________________________________________
varnish-misc mailing list
varnish-misc at varnish-cache.org<mailto:varnish-misc at varnish-cache.org>
https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/varnish-misc/attachments/20211026/934589f9/attachment-0001.html>

From Benny_Sham at macroview.com  Wed Oct 27 06:32:09 2021
From: Benny_Sham at macroview.com (Benny Sham)
Date: Wed, 27 Oct 2021 06:32:09 +0000
Subject: Empty varnishstat for Varnish 6.4.0
In-Reply-To: <f2e218c106724eea9fb758ebdb6f75d6@macroview.com>
References: <f2e218c106724eea9fb758ebdb6f75d6@macroview.com>
Message-ID: <98be0c9b7ee848648fcefddc3f1b10d9@macroview.com>

Hi,

I am trying to use "varnishstat -1", but it returns empty with exit code 0.

However, varnishadm (as follows) is working.

# varnishadm -T 127.0.0.1:6082 -S /etc/varnish/secret "help"

Below is some information.

Please let me know if more info is required. Thanks a lot.

---

# varnishd -V
varnishd (varnish-6.4.0 revision 13f137934ec1cf14af66baf7896311115ee35598)

# apt-cache policy varnish
varnish:
  Installed: 6.4.0-3
  Candidate: 6.4.0-3
  Version table:
 *** 6.4.0-3 500
   100 http://archive.ubuntu.com/ubuntu groovy/universe amd64 Packages
    100 /var/lib/dpkg/status

# cat /etc/systemd/system/varnish.service
...
ExecStart=/usr/sbin/varnishd -j unix,user=vcache -F -a :6081 -T localhost:6082 -f /etc/varnish/default.vcl -S /etc/varnish/secret -s default,256m
...

Regards,
Benny


________________________________

IMPORTANT NOTICE:
This e-mail (and any attachments) is confidential and may not be distributed, copied or used without the prior consent of Macroview Telecom Ltd., its affiliates or related bodies (?the Macroview Group?). The contents of this email may as well be legally privileged or otherwise copyright-protected. If you are not the intended recipient please notify us immediately by reply e-mail or telephone and delete this email (and any attachment) from your system, and destroy any printed copy. Any view or opinion expressed in this email may be those of the sender and not the Macroview Group.
Internet communications cannot be guaranteed to be timely, secure, error or virus-free. You should check for viruses before opening or using any attachments. We do not accept liability relating to computer virus, data corruption, delay, interruption, unauthorised access or unauthorised amendment.

From Benny_Sham at macroview.com  Wed Oct 27 08:31:16 2021
From: Benny_Sham at macroview.com (Benny Sham)
Date: Wed, 27 Oct 2021 08:31:16 +0000
Subject: Empty varnishstat for Varnish 6.4.0
In-Reply-To: <98be0c9b7ee848648fcefddc3f1b10d9@macroview.com>
References: <f2e218c106724eea9fb758ebdb6f75d6@macroview.com>
 <98be0c9b7ee848648fcefddc3f1b10d9@macroview.com>
Message-ID: <65289a11bc6b442a96ac4df7c420b86c@macroview.com>

Thanks to bjorn and simon in IRC,

It is caused by version mismatch:

varnishstat -> 6.2.1
varnishd -> 6.4.0

The problem is resolved.

Regards,
Benny

-----Original Message-----
From: Benny Sham
Sent: Wednesday, October 27, 2021 2:32 PM
To: 'varnish-misc at varnish-cache.org' <varnish-misc at varnish-cache.org>
Subject: Empty varnishstat for Varnish 6.4.0

Hi,

I am trying to use "varnishstat -1", but it returns empty with exit code 0.

However, varnishadm (as follows) is working.

# varnishadm -T 127.0.0.1:6082 -S /etc/varnish/secret "help"

Below is some information.

Please let me know if more info is required. Thanks a lot.

---

# varnishd -V
varnishd (varnish-6.4.0 revision 13f137934ec1cf14af66baf7896311115ee35598)

# apt-cache policy varnish
varnish:
  Installed: 6.4.0-3
  Candidate: 6.4.0-3
  Version table:
 *** 6.4.0-3 500
   100 http://archive.ubuntu.com/ubuntu groovy/universe amd64 Packages
    100 /var/lib/dpkg/status

# cat /etc/systemd/system/varnish.service
...
ExecStart=/usr/sbin/varnishd -j unix,user=vcache -F -a :6081 -T localhost:6082 -f /etc/varnish/default.vcl -S /etc/varnish/secret -s default,256m ...

Regards,
Benny


________________________________

IMPORTANT NOTICE:
This e-mail (and any attachments) is confidential and may not be distributed, copied or used without the prior consent of Macroview Telecom Ltd., its affiliates or related bodies (?the Macroview Group?). The contents of this email may as well be legally privileged or otherwise copyright-protected. If you are not the intended recipient please notify us immediately by reply e-mail or telephone and delete this email (and any attachment) from your system, and destroy any printed copy. Any view or opinion expressed in this email may be those of the sender and not the Macroview Group.
Internet communications cannot be guaranteed to be timely, secure, error or virus-free. You should check for viruses before opening or using any attachments. We do not accept liability relating to computer virus, data corruption, delay, interruption, unauthorised access or unauthorised amendment.