From phk at phk.freebsd.dk  Fri Apr 13 07:00:06 2018
From: phk at phk.freebsd.dk (Poul-Henning Kamp)
Date: Fri, 13 Apr 2018 07:00:06 +0000
Subject: Springtime for VDD ?
Message-ID: <58744.1523602806@critter.freebsd.dk>

A bugwash or two ago we talked about having a VDD "soonish",
did anybody do more about that ?

Does anybody want to do more about that ?

Poul-Henning

-- 
Poul-Henning Kamp       | UNIX since Zilog Zeus 3.20
phk at FreeBSD.ORG         | TCP/IP since RFC 956
FreeBSD committer       | BSD since 4.3-tahoe
Never attribute to malice what can adequately be explained by incompetence.

From fgsch at lodoss.net  Fri Apr 13 13:56:57 2018
From: fgsch at lodoss.net (Federico Schwindt)
Date: Fri, 13 Apr 2018 14:56:57 +0100
Subject: Springtime for VDD ?
In-Reply-To: <58744.1523602806@critter.freebsd.dk>
References: <58744.1523602806@critter.freebsd.dk>
Message-ID: <CAJV_h0aoAyxssZF7ZCFe1wXggiUd1p+-LHiKOHnjrdowN+6oow@mail.gmail.com>

Not sure about doing but providing there is enough notice, I'd love to
attend.

If there is anything I can do to help moving this forward please do let me
know.

On Fri, Apr 13, 2018 at 8:00 AM, Poul-Henning Kamp <phk at phk.freebsd.dk>
wrote:

> A bugwash or two ago we talked about having a VDD "soonish",
> did anybody do more about that ?
>
> Does anybody want to do more about that ?
>
> Poul-Henning
>
> --
> Poul-Henning Kamp       | UNIX since Zilog Zeus 3.20
> phk at FreeBSD.ORG         | TCP/IP since RFC 956
> FreeBSD committer       | BSD since 4.3-tahoe
> Never attribute to malice what can adequately be explained by incompetence.
> _______________________________________________
> varnish-dev mailing list
> varnish-dev at varnish-cache.org
> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/varnish-dev/attachments/20180413/df34860f/attachment.html>

From martin at varnish-software.com  Fri Apr 13 14:22:06 2018
From: martin at varnish-software.com (Martin Blix Grydeland)
Date: Fri, 13 Apr 2018 14:22:06 +0000
Subject: Springtime for VDD ?
In-Reply-To: <CAJV_h0aoAyxssZF7ZCFe1wXggiUd1p+-LHiKOHnjrdowN+6oow@mail.gmail.com>
References: <58744.1523602806@critter.freebsd.dk>
 <CAJV_h0aoAyxssZF7ZCFe1wXggiUd1p+-LHiKOHnjrdowN+6oow@mail.gmail.com>
Message-ID: <CANTn4cq1ayn2imURPL1oBJ4mOh_mnAATuTLoER6fMBFU+GqV6Q@mail.gmail.com>

We would be very happy to host a VDD here in Oslo. We have a medium sized
(8) meeting room available, and we can organize lunch in our canteen, plus
snacks to keep the blood sugar high during the day.

The tricky point is to agree on the dates I guess. The meeting room is
available when we want it, so that's easy. Our usual suspects are all back
in the office now, and the calendars look compatible with most dates
(though the waters are treacherous in May wrt holidays :)

To get things started, I will suggest Wednesday 9th of May. That is the day
before the Ascension-day, so assuming that is a holiday for everyone it
would be possible to spend an extra spring day in Oslo.

Martin



On Fri, 13 Apr 2018 at 15:57 Federico Schwindt <fgsch at lodoss.net> wrote:

> Not sure about doing but providing there is enough notice, I'd love to
> attend.
>
> If there is anything I can do to help moving this forward please do let me
> know.
>
> On Fri, Apr 13, 2018 at 8:00 AM, Poul-Henning Kamp <phk at phk.freebsd.dk>
> wrote:
>
>> A bugwash or two ago we talked about having a VDD "soonish",
>> did anybody do more about that ?
>>
>> Does anybody want to do more about that ?
>>
>> Poul-Henning
>>
>> --
>> Poul-Henning Kamp       | UNIX since Zilog Zeus 3.20
>> phk at FreeBSD.ORG         | TCP/IP since RFC 956
>> FreeBSD committer       | BSD since 4.3-tahoe
>> Never attribute to malice what can adequately be explained by
>> incompetence.
>> _______________________________________________
>> varnish-dev mailing list
>> varnish-dev at varnish-cache.org
>> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-dev
>>
>
> _______________________________________________
> varnish-dev mailing list
> varnish-dev at varnish-cache.org
> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-dev
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/varnish-dev/attachments/20180413/5445b059/attachment-0001.html>

From phk at phk.freebsd.dk  Fri Apr 13 14:29:19 2018
From: phk at phk.freebsd.dk (Poul-Henning Kamp)
Date: Fri, 13 Apr 2018 14:29:19 +0000
Subject: Springtime for VDD ?
In-Reply-To: <CANTn4cq1ayn2imURPL1oBJ4mOh_mnAATuTLoER6fMBFU+GqV6Q@mail.gmail.com>
References: <58744.1523602806@critter.freebsd.dk>
 <CAJV_h0aoAyxssZF7ZCFe1wXggiUd1p+-LHiKOHnjrdowN+6oow@mail.gmail.com>
 <CANTn4cq1ayn2imURPL1oBJ4mOh_mnAATuTLoER6fMBFU+GqV6Q@mail.gmail.com>
Message-ID: <61057.1523629759@critter.freebsd.dk>

--------
In message <CANTn4cq1ayn2imURPL1oBJ4mOh_mnAATuTLoER6fMBFU+GqV6Q at mail.gmail.com>
, Martin Blix Grydeland writes:

>To get things started, I will suggest Wednesday 9th of May. That is the day
>before the Ascension-day, so assuming that is a holiday for everyone it
>would be possible to spend an extra spring day in Oslo.

How about making a doodle.com with some possible dates, and lets see
what people say ?

-- 
Poul-Henning Kamp       | UNIX since Zilog Zeus 3.20
phk at FreeBSD.ORG         | TCP/IP since RFC 956
FreeBSD committer       | BSD since 4.3-tahoe    
Never attribute to malice what can adequately be explained by incompetence.

From dridi at varni.sh  Fri Apr 13 20:16:05 2018
From: dridi at varni.sh (Dridi Boukelmoune)
Date: Fri, 13 Apr 2018 22:16:05 +0200
Subject: Springtime for VDD ?
In-Reply-To: <58744.1523602806@critter.freebsd.dk>
References: <58744.1523602806@critter.freebsd.dk>
Message-ID: <CABoVN9CgHjCCp_2fa7n-DBDh45eKqYtyDiO7zF-h-gxMuKuH_w@mail.gmail.com>

On Fri, Apr 13, 2018 at 9:00 AM, Poul-Henning Kamp <phk at phk.freebsd.dk> wrote:
> A bugwash or two ago we talked about having a VDD "soonish",
> did anybody do more about that ?
>
> Does anybody want to do more about that ?

As I'm going to be away while the VDD is being organized, I'm dumping
here the topics I'd like to cover for the September release (this is
also a reminder for my future self).

First, actionable items I can/wish to deliver. I have discussed each
with at least one developer from VS and a VDD would be a good time to
quickly present them (as they are quite small in scope) to the rest:

- selinux
- extensions to the VSL query syntax
- (some) performance improvements for VSL consumers
- client side latency

That's in no particular order. If I can't make it to the VDD I'll
either start one thread for each on this list or bring them up during
bugwashes. As those have already been discussed, they shouldn't take
much time to cover. This is more about getting an OK before working on
them now that they have been fleshed out.

There's another topic I'd like to present around connection pools,
only to figure whether I'm not alone thinking this is a problem, and
also figure out new problems it could create. Once again if it doesn't
land on the VDD's agenda I'll either start a thread, open an issue or
bring it up during a bugwash.

Cheers,
Dridi

From phk at phk.freebsd.dk  Mon Apr 16 08:25:35 2018
From: phk at phk.freebsd.dk (Poul-Henning Kamp)
Date: Mon, 16 Apr 2018 08:25:35 +0000
Subject: Springtime for VDD ?
In-Reply-To: <CANTn4cq1ayn2imURPL1oBJ4mOh_mnAATuTLoER6fMBFU+GqV6Q@mail.gmail.com>
References: <58744.1523602806@critter.freebsd.dk>
 <CAJV_h0aoAyxssZF7ZCFe1wXggiUd1p+-LHiKOHnjrdowN+6oow@mail.gmail.com>
 <CANTn4cq1ayn2imURPL1oBJ4mOh_mnAATuTLoER6fMBFU+GqV6Q@mail.gmail.com>
Message-ID: <60611.1523867135@critter.freebsd.dk>

--------
In message <CANTn4cq1ayn2imURPL1oBJ4mOh_mnAATuTLoER6fMBFU+GqV6Q at mail.gmail.com>, Martin Blix Grydeland writes:

>To get things started, I will suggest Wednesday 9th of May.

As some of you may have heard, negotiations about public employees
wages are not going too well here in Denmark, and we may have what
amounts to a general strike on our hands pretty soon.  Amongst other
things, that will paralyze traffic, so I may not be able to get
anywhere if that happens.

I still think we should plan on a VDD meeting (see previous
email about a doodle) but it may be wise to not buy unrefundable
flight tickets until we know if I'm stuck in Denmark or not...

-- 
Poul-Henning Kamp       | UNIX since Zilog Zeus 3.20
phk at FreeBSD.ORG         | TCP/IP since RFC 956
FreeBSD committer       | BSD since 4.3-tahoe    
Never attribute to malice what can adequately be explained by incompetence.

From martin at varnish-software.com  Mon Apr 16 08:48:07 2018
From: martin at varnish-software.com (Martin Blix Grydeland)
Date: Mon, 16 Apr 2018 08:48:07 +0000
Subject: Springtime for VDD ?
In-Reply-To: <60611.1523867135@critter.freebsd.dk>
References: <58744.1523602806@critter.freebsd.dk>
 <CAJV_h0aoAyxssZF7ZCFe1wXggiUd1p+-LHiKOHnjrdowN+6oow@mail.gmail.com>
 <CANTn4cq1ayn2imURPL1oBJ4mOh_mnAATuTLoER6fMBFU+GqV6Q@mail.gmail.com>
 <60611.1523867135@critter.freebsd.dk>
Message-ID: <CANTn4cqJ4UF_s1VFNYA40nJf0x8JDx-xf7Cei_Rq-xn1ZyEEjA@mail.gmail.com>

I've created a doodle, putting all dates in May that are not public
holidays here.

https://doodle.com/poll/tbhgzk93p4x9hfx2

Martin

On Mon, 16 Apr 2018 at 10:25 Poul-Henning Kamp <phk at phk.freebsd.dk> wrote:

> --------
> In message <
> CANTn4cq1ayn2imURPL1oBJ4mOh_mnAATuTLoER6fMBFU+GqV6Q at mail.gmail.com>,
> Martin Blix Grydeland writes:
>
> >To get things started, I will suggest Wednesday 9th of May.
>
> As some of you may have heard, negotiations about public employees
> wages are not going too well here in Denmark, and we may have what
> amounts to a general strike on our hands pretty soon.  Amongst other
> things, that will paralyze traffic, so I may not be able to get
> anywhere if that happens.
>
> I still think we should plan on a VDD meeting (see previous
> email about a doodle) but it may be wise to not buy unrefundable
> flight tickets until we know if I'm stuck in Denmark or not...
>
> --
> Poul-Henning Kamp       | UNIX since Zilog Zeus 3.20
> phk at FreeBSD.ORG         | TCP/IP since RFC 956
> FreeBSD committer       | BSD since 4.3-tahoe
> Never attribute to malice what can adequately be explained by incompetence.
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/varnish-dev/attachments/20180416/1df27fc8/attachment.html>

From daghf at varnish-software.com  Tue Apr 17 11:32:39 2018
From: daghf at varnish-software.com (Dag Haavi Finstad)
Date: Tue, 17 Apr 2018 13:32:39 +0200
Subject: Springtime for VDD ?
In-Reply-To: <CANTn4cqJ4UF_s1VFNYA40nJf0x8JDx-xf7Cei_Rq-xn1ZyEEjA@mail.gmail.com>
References: <58744.1523602806@critter.freebsd.dk>
 <CAJV_h0aoAyxssZF7ZCFe1wXggiUd1p+-LHiKOHnjrdowN+6oow@mail.gmail.com>
 <CANTn4cq1ayn2imURPL1oBJ4mOh_mnAATuTLoER6fMBFU+GqV6Q@mail.gmail.com>
 <60611.1523867135@critter.freebsd.dk>
 <CANTn4cqJ4UF_s1VFNYA40nJf0x8JDx-xf7Cei_Rq-xn1ZyEEjA@mail.gmail.com>
Message-ID: <CAEh05VZnK19w2ruY=S--h_HtPgzBw0E_b3qwAaRE_w+zzU=c_g@mail.gmail.com>

The date has been set: Monday May 14th

On Mon, Apr 16, 2018 at 10:48 AM, Martin Blix Grydeland <
martin at varnish-software.com> wrote:

> I've created a doodle, putting all dates in May that are not public
> holidays here.
>
> https://doodle.com/poll/tbhgzk93p4x9hfx2
>
> Martin
>
> On Mon, 16 Apr 2018 at 10:25 Poul-Henning Kamp <phk at phk.freebsd.dk> wrote:
>
>> --------
>> In message <CANTn4cq1ayn2imURPL1oBJ4mOh_mnAATuTLoER6fMBFU+GqV6Q at mail.
>> gmail.com>, Martin Blix Grydeland writes:
>>
>> >To get things started, I will suggest Wednesday 9th of May.
>>
>> As some of you may have heard, negotiations about public employees
>> wages are not going too well here in Denmark, and we may have what
>> amounts to a general strike on our hands pretty soon.  Amongst other
>> things, that will paralyze traffic, so I may not be able to get
>> anywhere if that happens.
>>
>> I still think we should plan on a VDD meeting (see previous
>> email about a doodle) but it may be wise to not buy unrefundable
>> flight tickets until we know if I'm stuck in Denmark or not...
>>
>> --
>> Poul-Henning Kamp       | UNIX since Zilog Zeus 3.20
>> phk at FreeBSD.ORG         | TCP/IP since RFC 956
>> FreeBSD committer       | BSD since 4.3-tahoe
>> Never attribute to malice what can adequately be explained by
>> incompetence.
>>
>
> _______________________________________________
> varnish-dev mailing list
> varnish-dev at varnish-cache.org
> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-dev
>



-- 
Dag Haavi Finstad
Software Developer | Varnish Software
Mobile: +47 476 64 134
We Make Websites Fly!
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/varnish-dev/attachments/20180417/ffb79111/attachment.html>

From stephen.butler at gmail.com  Fri Apr 27 06:05:56 2018
From: stephen.butler at gmail.com (Stephen J. Butler)
Date: Fri, 27 Apr 2018 01:05:56 -0500
Subject: PRIV_CALL, VRT_re_init, and mutexes
Message-ID: <CAD4ANxW-hyHmrURCHDN2ZQ9aJYhNT5+dCE+Tuq+4jnB1W2MA8Q@mail.gmail.com>

I'm a new developer/user for Varnish, and was looking to extend a module
with some regex support (vmod_cookie if you must know). I don't know
anything about vmod development or Varnish threading, so to get an idea of
how to do this I looked at the vmod_header module.

But I'm confused about why they thought they needed a mutex to protected a
call to VRT_re_init. Is this just a useless, historic thing that got left
in for some reason?

https://github.com/varnish/varnish-modules/blob/master/src/vmod_header.c

If you look at vmod_remove it has a 2nd parameter that's in the vcc as
PRIV_CALL. This parameter is initialized with the string value from the
last parameter (a regex pattern). It does this by calling header_init_re().
And if you look at that function you see:

/*
* Initialize the regex *s on priv, if it hasn't already been done.
* XXX: We have to recheck the condition after grabbing the lock to avoid a
* XXX: race condition.
*/
static void
header_init_re(struct vmod_priv *priv, const char *s)
{
if (priv->priv == NULL) {
assert(pthread_mutex_lock(&header_mutex) == 0);
if (priv->priv == NULL) {
VRT_re_init(&priv->priv, s);
priv->free = VRT_re_fini;
}
pthread_mutex_unlock(&header_mutex);
}
}
My reading of the docs makes me think that this "priv" only lives for each
single call of the function. There should be no reason to protect it with a
global mutex. Do the internals of VRT_re_init() require this? I can't
imagine how, because if it did then this method of protecting it is broken
anyway.

I suspect this is leftover from some rewrite or updating of the module but
wanted to check.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/varnish-dev/attachments/20180427/27bd57c5/attachment-0001.html>

From guillaume at varnish-software.com  Fri Apr 27 06:37:21 2018
From: guillaume at varnish-software.com (Guillaume Quintard)
Date: Fri, 27 Apr 2018 08:37:21 +0200
Subject: PRIV_CALL, VRT_re_init, and mutexes
In-Reply-To: <CAD4ANxW-hyHmrURCHDN2ZQ9aJYhNT5+dCE+Tuq+4jnB1W2MA8Q@mail.gmail.com>
References: <CAD4ANxW-hyHmrURCHDN2ZQ9aJYhNT5+dCE+Tuq+4jnB1W2MA8Q@mail.gmail.com>
Message-ID: <CAJ6ZYQy+iTpzQY_Ps_ZwgJxUy07yt49ay52OVWzq9aF-Hn6zfg@mail.gmail.com>

Hi, and welcome!

In this case, priv comes from PRIV_CALL, meaning it's going to be the same
for all the request/vcl that call the function from the same spot. This is
done to cache the regex, so you only compile for the first call.

However, I'm thinking that you may have the same priv with a different
string, and then everything goes to hell. Could anyone familiar with
PRIV_CALL confirm?

Cheers,

-- 
Guillaume Quintard

On Fri, Apr 27, 2018 at 8:05 AM, Stephen J. Butler <stephen.butler at gmail.com
> wrote:

> I'm a new developer/user for Varnish, and was looking to extend a module
> with some regex support (vmod_cookie if you must know). I don't know
> anything about vmod development or Varnish threading, so to get an idea of
> how to do this I looked at the vmod_header module.
>
> But I'm confused about why they thought they needed a mutex to protected a
> call to VRT_re_init. Is this just a useless, historic thing that got left
> in for some reason?
>
> https://github.com/varnish/varnish-modules/blob/master/src/vmod_header.c
>
> If you look at vmod_remove it has a 2nd parameter that's in the vcc as
> PRIV_CALL. This parameter is initialized with the string value from the
> last parameter (a regex pattern). It does this by calling header_init_re().
> And if you look at that function you see:
>
> /*
> * Initialize the regex *s on priv, if it hasn't already been done.
> * XXX: We have to recheck the condition after grabbing the lock to avoid a
> * XXX: race condition.
> */
> static void
> header_init_re(struct vmod_priv *priv, const char *s)
> {
> if (priv->priv == NULL) {
> assert(pthread_mutex_lock(&header_mutex) == 0);
> if (priv->priv == NULL) {
> VRT_re_init(&priv->priv, s);
> priv->free = VRT_re_fini;
> }
> pthread_mutex_unlock(&header_mutex);
> }
> }
> My reading of the docs makes me think that this "priv" only lives for each
> single call of the function. There should be no reason to protect it with a
> global mutex. Do the internals of VRT_re_init() require this? I can't
> imagine how, because if it did then this method of protecting it is broken
> anyway.
>
> I suspect this is leftover from some rewrite or updating of the module but
> wanted to check.
>
>
> _______________________________________________
> varnish-dev mailing list
> varnish-dev at varnish-cache.org
> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/varnish-dev/attachments/20180427/ba439f5b/attachment.html>

From stephen.butler at gmail.com  Fri Apr 27 06:49:05 2018
From: stephen.butler at gmail.com (Stephen J. Butler)
Date: Fri, 27 Apr 2018 01:49:05 -0500
Subject: PRIV_CALL, VRT_re_init, and mutexes
In-Reply-To: <CAD4ANxV0dQiwTgcXyE3M8ZCFeQY_v9_F_Vy_L=EUOu0D-JQ2jQ@mail.gmail.com>
References: <CAD4ANxW-hyHmrURCHDN2ZQ9aJYhNT5+dCE+Tuq+4jnB1W2MA8Q@mail.gmail.com>
 <CAJ6ZYQy+iTpzQY_Ps_ZwgJxUy07yt49ay52OVWzq9aF-Hn6zfg@mail.gmail.com>
 <CAD4ANxV0dQiwTgcXyE3M8ZCFeQY_v9_F_Vy_L=EUOu0D-JQ2jQ@mail.gmail.com>
Message-ID: <CAD4ANxW_b+h=VortUk-+uNVih7cim2sFkdcoecT9y-FVWw70Mg@mail.gmail.com>

Sorry, didn't reply to the list:

On Fri, Apr 27, 2018 at 1:48 AM, Stephen J. Butler <stephen.butler at gmail.com
> wrote:

> Hmm. If PRIV_CALL is private to the call site, then I think
> in vmod_bodyaccess.c, vmod_rematch_req_body()there's a problem. In that
> case it calls VRE_compile()to initiate a regex without a lock on a
> PRIV_CALL structure.
>
> On Fri, Apr 27, 2018 at 1:37 AM, Guillaume Quintard <
> guillaume at varnish-software.com> wrote:
>
>> Hi, and welcome!
>>
>> In this case, priv comes from PRIV_CALL, meaning it's going to be the
>> same for all the request/vcl that call the function from the same spot.
>> This is done to cache the regex, so you only compile for the first call.
>>
>> However, I'm thinking that you may have the same priv with a different
>> string, and then everything goes to hell. Could anyone familiar with
>> PRIV_CALL confirm?
>>
>> Cheers,
>>
>> --
>> Guillaume Quintard
>>
>> On Fri, Apr 27, 2018 at 8:05 AM, Stephen J. Butler <
>> stephen.butler at gmail.com> wrote:
>>
>>> I'm a new developer/user for Varnish, and was looking to extend a module
>>> with some regex support (vmod_cookie if you must know). I don't know
>>> anything about vmod development or Varnish threading, so to get an idea of
>>> how to do this I looked at the vmod_header module.
>>>
>>> But I'm confused about why they thought they needed a mutex to protected
>>> a call to VRT_re_init. Is this just a useless, historic thing that got left
>>> in for some reason?
>>>
>>> https://github.com/varnish/varnish-modules/blob/master/src/vmod_header.c
>>>
>>> If you look at vmod_remove it has a 2nd parameter that's in the vcc as
>>> PRIV_CALL. This parameter is initialized with the string value from the
>>> last parameter (a regex pattern). It does this by calling
>>> header_init_re(). And if you look at that function you see:
>>>
>>> /*
>>> * Initialize the regex *s on priv, if it hasn't already been done.
>>> * XXX: We have to recheck the condition after grabbing the lock to avoid
>>> a
>>> * XXX: race condition.
>>> */
>>> static void
>>> header_init_re(struct vmod_priv *priv, const char *s)
>>> {
>>> if (priv->priv == NULL) {
>>> assert(pthread_mutex_lock(&header_mutex) == 0);
>>> if (priv->priv == NULL) {
>>> VRT_re_init(&priv->priv, s);
>>> priv->free = VRT_re_fini;
>>> }
>>> pthread_mutex_unlock(&header_mutex);
>>> }
>>> }
>>> My reading of the docs makes me think that this "priv" only lives for
>>> each single call of the function. There should be no reason to protect it
>>> with a global mutex. Do the internals of VRT_re_init() require this? I
>>> can't imagine how, because if it did then this method of protecting it is
>>> broken anyway.
>>>
>>> I suspect this is leftover from some rewrite or updating of the module
>>> but wanted to check.
>>>
>>>
>>> _______________________________________________
>>> varnish-dev mailing list
>>> varnish-dev at varnish-cache.org
>>> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-dev
>>>
>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/varnish-dev/attachments/20180427/020e5cb4/attachment-0001.html>

From stephen.butler at gmail.com  Fri Apr 27 07:05:28 2018
From: stephen.butler at gmail.com (Stephen J. Butler)
Date: Fri, 27 Apr 2018 02:05:28 -0500
Subject: PRIV_CALL, VRT_re_init, and mutexes
In-Reply-To: <CAD4ANxW_b+h=VortUk-+uNVih7cim2sFkdcoecT9y-FVWw70Mg@mail.gmail.com>
References: <CAD4ANxW-hyHmrURCHDN2ZQ9aJYhNT5+dCE+Tuq+4jnB1W2MA8Q@mail.gmail.com>
 <CAJ6ZYQy+iTpzQY_Ps_ZwgJxUy07yt49ay52OVWzq9aF-Hn6zfg@mail.gmail.com>
 <CAD4ANxV0dQiwTgcXyE3M8ZCFeQY_v9_F_Vy_L=EUOu0D-JQ2jQ@mail.gmail.com>
 <CAD4ANxW_b+h=VortUk-+uNVih7cim2sFkdcoecT9y-FVWw70Mg@mail.gmail.com>
Message-ID: <CAD4ANxXNh65AjNmsEVsWRMGFtPfxm3+fsURud91EHQUTUesFcw@mail.gmail.com>

In the varnish source code for the 6.0 release, I found one use of
PRIV_CALL in vmod_std, vmod_fileread(). This function does have locks, but
they're all to protect modifications of the global frlist variable. It does
seem like the priv parameter is used as a call site value but no care is
taken to make sure access is exclusive.

In that case it also bothers me that there's an assumption that -- and ++
are atomic operations on all platforms and compilers... some quick Googling
suggests that's not the case.


On Fri, Apr 27, 2018 at 1:49 AM, Stephen J. Butler <stephen.butler at gmail.com
> wrote:

> Sorry, didn't reply to the list:
>
> On Fri, Apr 27, 2018 at 1:48 AM, Stephen J. Butler <
> stephen.butler at gmail.com> wrote:
>
>> Hmm. If PRIV_CALL is private to the call site, then I think
>> in vmod_bodyaccess.c, vmod_rematch_req_body()there's a problem. In that
>> case it calls VRE_compile()to initiate a regex without a lock on a
>> PRIV_CALL structure.
>>
>> On Fri, Apr 27, 2018 at 1:37 AM, Guillaume Quintard <
>> guillaume at varnish-software.com> wrote:
>>
>>> Hi, and welcome!
>>>
>>> In this case, priv comes from PRIV_CALL, meaning it's going to be the
>>> same for all the request/vcl that call the function from the same spot.
>>> This is done to cache the regex, so you only compile for the first call.
>>>
>>> However, I'm thinking that you may have the same priv with a different
>>> string, and then everything goes to hell. Could anyone familiar with
>>> PRIV_CALL confirm?
>>>
>>> Cheers,
>>>
>>> --
>>> Guillaume Quintard
>>>
>>> On Fri, Apr 27, 2018 at 8:05 AM, Stephen J. Butler <
>>> stephen.butler at gmail.com> wrote:
>>>
>>>> I'm a new developer/user for Varnish, and was looking to extend a
>>>> module with some regex support (vmod_cookie if you must know). I don't know
>>>> anything about vmod development or Varnish threading, so to get an idea of
>>>> how to do this I looked at the vmod_header module.
>>>>
>>>> But I'm confused about why they thought they needed a mutex to
>>>> protected a call to VRT_re_init. Is this just a useless, historic thing
>>>> that got left in for some reason?
>>>>
>>>> https://github.com/varnish/varnish-modules/blob/master/src/v
>>>> mod_header.c
>>>>
>>>> If you look at vmod_remove it has a 2nd parameter that's in the vcc as
>>>> PRIV_CALL. This parameter is initialized with the string value from the
>>>> last parameter (a regex pattern). It does this by calling
>>>> header_init_re(). And if you look at that function you see:
>>>>
>>>> /*
>>>> * Initialize the regex *s on priv, if it hasn't already been done.
>>>> * XXX: We have to recheck the condition after grabbing the lock to
>>>> avoid a
>>>> * XXX: race condition.
>>>> */
>>>> static void
>>>> header_init_re(struct vmod_priv *priv, const char *s)
>>>> {
>>>> if (priv->priv == NULL) {
>>>> assert(pthread_mutex_lock(&header_mutex) == 0);
>>>> if (priv->priv == NULL) {
>>>> VRT_re_init(&priv->priv, s);
>>>> priv->free = VRT_re_fini;
>>>> }
>>>> pthread_mutex_unlock(&header_mutex);
>>>> }
>>>> }
>>>> My reading of the docs makes me think that this "priv" only lives for
>>>> each single call of the function. There should be no reason to protect it
>>>> with a global mutex. Do the internals of VRT_re_init() require this? I
>>>> can't imagine how, because if it did then this method of protecting it is
>>>> broken anyway.
>>>>
>>>> I suspect this is leftover from some rewrite or updating of the module
>>>> but wanted to check.
>>>>
>>>>
>>>> _______________________________________________
>>>> varnish-dev mailing list
>>>> varnish-dev at varnish-cache.org
>>>> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-dev
>>>>
>>>
>>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/varnish-dev/attachments/20180427/b470d0f4/attachment-0001.html>

From stephen.butler at gmail.com  Fri Apr 27 07:06:46 2018
From: stephen.butler at gmail.com (Stephen J. Butler)
Date: Fri, 27 Apr 2018 02:06:46 -0500
Subject: PRIV_CALL, VRT_re_init, and mutexes
In-Reply-To: <CAD4ANxXNh65AjNmsEVsWRMGFtPfxm3+fsURud91EHQUTUesFcw@mail.gmail.com>
References: <CAD4ANxW-hyHmrURCHDN2ZQ9aJYhNT5+dCE+Tuq+4jnB1W2MA8Q@mail.gmail.com>
 <CAJ6ZYQy+iTpzQY_Ps_ZwgJxUy07yt49ay52OVWzq9aF-Hn6zfg@mail.gmail.com>
 <CAD4ANxV0dQiwTgcXyE3M8ZCFeQY_v9_F_Vy_L=EUOu0D-JQ2jQ@mail.gmail.com>
 <CAD4ANxW_b+h=VortUk-+uNVih7cim2sFkdcoecT9y-FVWw70Mg@mail.gmail.com>
 <CAD4ANxXNh65AjNmsEVsWRMGFtPfxm3+fsURud91EHQUTUesFcw@mail.gmail.com>
Message-ID: <CAD4ANxXDzenKBSAXNsJ-xQyGDW-uJMGsFcSZZpkphbeiZdeTjw@mail.gmail.com>

(sorry, never mind about the -- and ++; they're all inside the lock)

On Fri, Apr 27, 2018 at 2:05 AM, Stephen J. Butler <stephen.butler at gmail.com
> wrote:

> In the varnish source code for the 6.0 release, I found one use of
> PRIV_CALL in vmod_std, vmod_fileread(). This function does have locks, but
> they're all to protect modifications of the global frlist variable. It does
> seem like the priv parameter is used as a call site value but no care is
> taken to make sure access is exclusive.
>
> In that case it also bothers me that there's an assumption that -- and ++
> are atomic operations on all platforms and compilers... some quick Googling
> suggests that's not the case.
>
>
> On Fri, Apr 27, 2018 at 1:49 AM, Stephen J. Butler <
> stephen.butler at gmail.com> wrote:
>
>> Sorry, didn't reply to the list:
>>
>> On Fri, Apr 27, 2018 at 1:48 AM, Stephen J. Butler <
>> stephen.butler at gmail.com> wrote:
>>
>>> Hmm. If PRIV_CALL is private to the call site, then I think
>>> in vmod_bodyaccess.c, vmod_rematch_req_body()there's a problem. In that
>>> case it calls VRE_compile()to initiate a regex without a lock on a
>>> PRIV_CALL structure.
>>>
>>> On Fri, Apr 27, 2018 at 1:37 AM, Guillaume Quintard <
>>> guillaume at varnish-software.com> wrote:
>>>
>>>> Hi, and welcome!
>>>>
>>>> In this case, priv comes from PRIV_CALL, meaning it's going to be the
>>>> same for all the request/vcl that call the function from the same spot.
>>>> This is done to cache the regex, so you only compile for the first call.
>>>>
>>>> However, I'm thinking that you may have the same priv with a different
>>>> string, and then everything goes to hell. Could anyone familiar with
>>>> PRIV_CALL confirm?
>>>>
>>>> Cheers,
>>>>
>>>> --
>>>> Guillaume Quintard
>>>>
>>>> On Fri, Apr 27, 2018 at 8:05 AM, Stephen J. Butler <
>>>> stephen.butler at gmail.com> wrote:
>>>>
>>>>> I'm a new developer/user for Varnish, and was looking to extend a
>>>>> module with some regex support (vmod_cookie if you must know). I don't know
>>>>> anything about vmod development or Varnish threading, so to get an idea of
>>>>> how to do this I looked at the vmod_header module.
>>>>>
>>>>> But I'm confused about why they thought they needed a mutex to
>>>>> protected a call to VRT_re_init. Is this just a useless, historic thing
>>>>> that got left in for some reason?
>>>>>
>>>>> https://github.com/varnish/varnish-modules/blob/master/src/v
>>>>> mod_header.c
>>>>>
>>>>> If you look at vmod_remove it has a 2nd parameter that's in the vcc as
>>>>> PRIV_CALL. This parameter is initialized with the string value from the
>>>>> last parameter (a regex pattern). It does this by calling
>>>>> header_init_re(). And if you look at that function you see:
>>>>>
>>>>> /*
>>>>> * Initialize the regex *s on priv, if it hasn't already been done.
>>>>> * XXX: We have to recheck the condition after grabbing the lock to
>>>>> avoid a
>>>>> * XXX: race condition.
>>>>> */
>>>>> static void
>>>>> header_init_re(struct vmod_priv *priv, const char *s)
>>>>> {
>>>>> if (priv->priv == NULL) {
>>>>> assert(pthread_mutex_lock(&header_mutex) == 0);
>>>>> if (priv->priv == NULL) {
>>>>> VRT_re_init(&priv->priv, s);
>>>>> priv->free = VRT_re_fini;
>>>>> }
>>>>> pthread_mutex_unlock(&header_mutex);
>>>>> }
>>>>> }
>>>>> My reading of the docs makes me think that this "priv" only lives for
>>>>> each single call of the function. There should be no reason to protect it
>>>>> with a global mutex. Do the internals of VRT_re_init() require this? I
>>>>> can't imagine how, because if it did then this method of protecting it is
>>>>> broken anyway.
>>>>>
>>>>> I suspect this is leftover from some rewrite or updating of the module
>>>>> but wanted to check.
>>>>>
>>>>>
>>>>> _______________________________________________
>>>>> varnish-dev mailing list
>>>>> varnish-dev at varnish-cache.org
>>>>> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-dev
>>>>>
>>>>
>>>>
>>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/varnish-dev/attachments/20180427/2db5dd5a/attachment-0001.html>

From stephen.butler at gmail.com  Fri Apr 27 07:47:22 2018
From: stephen.butler at gmail.com (Stephen J. Butler)
Date: Fri, 27 Apr 2018 02:47:22 -0500
Subject: PRIV_CALL, VRT_re_init, and mutexes
In-Reply-To: <CAD4ANxXDzenKBSAXNsJ-xQyGDW-uJMGsFcSZZpkphbeiZdeTjw@mail.gmail.com>
References: <CAD4ANxW-hyHmrURCHDN2ZQ9aJYhNT5+dCE+Tuq+4jnB1W2MA8Q@mail.gmail.com>
 <CAJ6ZYQy+iTpzQY_Ps_ZwgJxUy07yt49ay52OVWzq9aF-Hn6zfg@mail.gmail.com>
 <CAD4ANxV0dQiwTgcXyE3M8ZCFeQY_v9_F_Vy_L=EUOu0D-JQ2jQ@mail.gmail.com>
 <CAD4ANxW_b+h=VortUk-+uNVih7cim2sFkdcoecT9y-FVWw70Mg@mail.gmail.com>
 <CAD4ANxXNh65AjNmsEVsWRMGFtPfxm3+fsURud91EHQUTUesFcw@mail.gmail.com>
 <CAD4ANxXDzenKBSAXNsJ-xQyGDW-uJMGsFcSZZpkphbeiZdeTjw@mail.gmail.com>
Message-ID: <CAD4ANxVFJtE+E_kyPSL4Yaq1Kj7cSbpCj-4Z_o05pmQi0ARtCg@mail.gmail.com>

OK, I think I figured it out. PRIV_CALL is a call site specific storage. I
ended up figuring this out by looking at the compiled VCL output from
varnishd.

For headers there should probably be documentation to not use a variable
argument for the pattern. In vmod_bodyaccess the same thing (no variables
for patterns) but it should also lock around the VRE_compile, and the free
function is incorrect and should be VRE_free, or it leaks memory during
race conditions.

On Fri, Apr 27, 2018 at 2:06 AM, Stephen J. Butler <stephen.butler at gmail.com
> wrote:

> (sorry, never mind about the -- and ++; they're all inside the lock)
>
> On Fri, Apr 27, 2018 at 2:05 AM, Stephen J. Butler <
> stephen.butler at gmail.com> wrote:
>
>> In the varnish source code for the 6.0 release, I found one use of
>> PRIV_CALL in vmod_std, vmod_fileread(). This function does have locks, but
>> they're all to protect modifications of the global frlist variable. It does
>> seem like the priv parameter is used as a call site value but no care is
>> taken to make sure access is exclusive.
>>
>> In that case it also bothers me that there's an assumption that -- and ++
>> are atomic operations on all platforms and compilers... some quick Googling
>> suggests that's not the case.
>>
>>
>> On Fri, Apr 27, 2018 at 1:49 AM, Stephen J. Butler <
>> stephen.butler at gmail.com> wrote:
>>
>>> Sorry, didn't reply to the list:
>>>
>>> On Fri, Apr 27, 2018 at 1:48 AM, Stephen J. Butler <
>>> stephen.butler at gmail.com> wrote:
>>>
>>>> Hmm. If PRIV_CALL is private to the call site, then I think
>>>> in vmod_bodyaccess.c, vmod_rematch_req_body()there's a problem. In
>>>> that case it calls VRE_compile()to initiate a regex without a lock on
>>>> a PRIV_CALL structure.
>>>>
>>>> On Fri, Apr 27, 2018 at 1:37 AM, Guillaume Quintard <
>>>> guillaume at varnish-software.com> wrote:
>>>>
>>>>> Hi, and welcome!
>>>>>
>>>>> In this case, priv comes from PRIV_CALL, meaning it's going to be the
>>>>> same for all the request/vcl that call the function from the same spot.
>>>>> This is done to cache the regex, so you only compile for the first call.
>>>>>
>>>>> However, I'm thinking that you may have the same priv with a different
>>>>> string, and then everything goes to hell. Could anyone familiar with
>>>>> PRIV_CALL confirm?
>>>>>
>>>>> Cheers,
>>>>>
>>>>> --
>>>>> Guillaume Quintard
>>>>>
>>>>> On Fri, Apr 27, 2018 at 8:05 AM, Stephen J. Butler <
>>>>> stephen.butler at gmail.com> wrote:
>>>>>
>>>>>> I'm a new developer/user for Varnish, and was looking to extend a
>>>>>> module with some regex support (vmod_cookie if you must know). I don't know
>>>>>> anything about vmod development or Varnish threading, so to get an idea of
>>>>>> how to do this I looked at the vmod_header module.
>>>>>>
>>>>>> But I'm confused about why they thought they needed a mutex to
>>>>>> protected a call to VRT_re_init. Is this just a useless, historic thing
>>>>>> that got left in for some reason?
>>>>>>
>>>>>> https://github.com/varnish/varnish-modules/blob/master/src/v
>>>>>> mod_header.c
>>>>>>
>>>>>> If you look at vmod_remove it has a 2nd parameter that's in the vcc
>>>>>> as PRIV_CALL. This parameter is initialized with the string value from the
>>>>>> last parameter (a regex pattern). It does this by calling
>>>>>> header_init_re(). And if you look at that function you see:
>>>>>>
>>>>>> /*
>>>>>> * Initialize the regex *s on priv, if it hasn't already been done.
>>>>>> * XXX: We have to recheck the condition after grabbing the lock to
>>>>>> avoid a
>>>>>> * XXX: race condition.
>>>>>> */
>>>>>> static void
>>>>>> header_init_re(struct vmod_priv *priv, const char *s)
>>>>>> {
>>>>>> if (priv->priv == NULL) {
>>>>>> assert(pthread_mutex_lock(&header_mutex) == 0);
>>>>>> if (priv->priv == NULL) {
>>>>>> VRT_re_init(&priv->priv, s);
>>>>>> priv->free = VRT_re_fini;
>>>>>> }
>>>>>> pthread_mutex_unlock(&header_mutex);
>>>>>> }
>>>>>> }
>>>>>> My reading of the docs makes me think that this "priv" only lives for
>>>>>> each single call of the function. There should be no reason to protect it
>>>>>> with a global mutex. Do the internals of VRT_re_init() require this? I
>>>>>> can't imagine how, because if it did then this method of protecting it is
>>>>>> broken anyway.
>>>>>>
>>>>>> I suspect this is leftover from some rewrite or updating of the
>>>>>> module but wanted to check.
>>>>>>
>>>>>>
>>>>>> _______________________________________________
>>>>>> varnish-dev mailing list
>>>>>> varnish-dev at varnish-cache.org
>>>>>> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-dev
>>>>>>
>>>>>
>>>>>
>>>>
>>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/varnish-dev/attachments/20180427/1e63cda4/attachment-0001.html>