From phk at phk.freebsd.dk Fri Apr 13 07:00:06 2018 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Fri, 13 Apr 2018 07:00:06 +0000 Subject: Springtime for VDD ? Message-ID: <58744.1523602806@critter.freebsd.dk> A bugwash or two ago we talked about having a VDD "soonish", did anybody do more about that ? Does anybody want to do more about that ? Poul-Henning -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From fgsch at lodoss.net Fri Apr 13 13:56:57 2018 From: fgsch at lodoss.net (Federico Schwindt) Date: Fri, 13 Apr 2018 14:56:57 +0100 Subject: Springtime for VDD ? In-Reply-To: <58744.1523602806@critter.freebsd.dk> References: <58744.1523602806@critter.freebsd.dk> Message-ID: Not sure about doing but providing there is enough notice, I'd love to attend. If there is anything I can do to help moving this forward please do let me know. On Fri, Apr 13, 2018 at 8:00 AM, Poul-Henning Kamp wrote: > A bugwash or two ago we talked about having a VDD "soonish", > did anybody do more about that ? > > Does anybody want to do more about that ? > > Poul-Henning > > -- > Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 > phk at FreeBSD.ORG | TCP/IP since RFC 956 > FreeBSD committer | BSD since 4.3-tahoe > Never attribute to malice what can adequately be explained by incompetence. > _______________________________________________ > varnish-dev mailing list > varnish-dev at varnish-cache.org > https://www.varnish-cache.org/lists/mailman/listinfo/varnish-dev > -------------- next part -------------- An HTML attachment was scrubbed... URL: From martin at varnish-software.com Fri Apr 13 14:22:06 2018 From: martin at varnish-software.com (Martin Blix Grydeland) Date: Fri, 13 Apr 2018 14:22:06 +0000 Subject: Springtime for VDD ? In-Reply-To: References: <58744.1523602806@critter.freebsd.dk> Message-ID: We would be very happy to host a VDD here in Oslo. We have a medium sized (8) meeting room available, and we can organize lunch in our canteen, plus snacks to keep the blood sugar high during the day. The tricky point is to agree on the dates I guess. The meeting room is available when we want it, so that's easy. Our usual suspects are all back in the office now, and the calendars look compatible with most dates (though the waters are treacherous in May wrt holidays :) To get things started, I will suggest Wednesday 9th of May. That is the day before the Ascension-day, so assuming that is a holiday for everyone it would be possible to spend an extra spring day in Oslo. Martin On Fri, 13 Apr 2018 at 15:57 Federico Schwindt wrote: > Not sure about doing but providing there is enough notice, I'd love to > attend. > > If there is anything I can do to help moving this forward please do let me > know. > > On Fri, Apr 13, 2018 at 8:00 AM, Poul-Henning Kamp > wrote: > >> A bugwash or two ago we talked about having a VDD "soonish", >> did anybody do more about that ? >> >> Does anybody want to do more about that ? >> >> Poul-Henning >> >> -- >> Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 >> phk at FreeBSD.ORG | TCP/IP since RFC 956 >> FreeBSD committer | BSD since 4.3-tahoe >> Never attribute to malice what can adequately be explained by >> incompetence. >> _______________________________________________ >> varnish-dev mailing list >> varnish-dev at varnish-cache.org >> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-dev >> > > _______________________________________________ > varnish-dev mailing list > varnish-dev at varnish-cache.org > https://www.varnish-cache.org/lists/mailman/listinfo/varnish-dev -------------- next part -------------- An HTML attachment was scrubbed... URL: From phk at phk.freebsd.dk Fri Apr 13 14:29:19 2018 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Fri, 13 Apr 2018 14:29:19 +0000 Subject: Springtime for VDD ? In-Reply-To: References: <58744.1523602806@critter.freebsd.dk> Message-ID: <61057.1523629759@critter.freebsd.dk> -------- In message , Martin Blix Grydeland writes: >To get things started, I will suggest Wednesday 9th of May. That is the day >before the Ascension-day, so assuming that is a holiday for everyone it >would be possible to spend an extra spring day in Oslo. How about making a doodle.com with some possible dates, and lets see what people say ? -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From dridi at varni.sh Fri Apr 13 20:16:05 2018 From: dridi at varni.sh (Dridi Boukelmoune) Date: Fri, 13 Apr 2018 22:16:05 +0200 Subject: Springtime for VDD ? In-Reply-To: <58744.1523602806@critter.freebsd.dk> References: <58744.1523602806@critter.freebsd.dk> Message-ID: On Fri, Apr 13, 2018 at 9:00 AM, Poul-Henning Kamp wrote: > A bugwash or two ago we talked about having a VDD "soonish", > did anybody do more about that ? > > Does anybody want to do more about that ? As I'm going to be away while the VDD is being organized, I'm dumping here the topics I'd like to cover for the September release (this is also a reminder for my future self). First, actionable items I can/wish to deliver. I have discussed each with at least one developer from VS and a VDD would be a good time to quickly present them (as they are quite small in scope) to the rest: - selinux - extensions to the VSL query syntax - (some) performance improvements for VSL consumers - client side latency That's in no particular order. If I can't make it to the VDD I'll either start one thread for each on this list or bring them up during bugwashes. As those have already been discussed, they shouldn't take much time to cover. This is more about getting an OK before working on them now that they have been fleshed out. There's another topic I'd like to present around connection pools, only to figure whether I'm not alone thinking this is a problem, and also figure out new problems it could create. Once again if it doesn't land on the VDD's agenda I'll either start a thread, open an issue or bring it up during a bugwash. Cheers, Dridi From phk at phk.freebsd.dk Mon Apr 16 08:25:35 2018 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Mon, 16 Apr 2018 08:25:35 +0000 Subject: Springtime for VDD ? In-Reply-To: References: <58744.1523602806@critter.freebsd.dk> Message-ID: <60611.1523867135@critter.freebsd.dk> -------- In message , Martin Blix Grydeland writes: >To get things started, I will suggest Wednesday 9th of May. As some of you may have heard, negotiations about public employees wages are not going too well here in Denmark, and we may have what amounts to a general strike on our hands pretty soon. Amongst other things, that will paralyze traffic, so I may not be able to get anywhere if that happens. I still think we should plan on a VDD meeting (see previous email about a doodle) but it may be wise to not buy unrefundable flight tickets until we know if I'm stuck in Denmark or not... -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From martin at varnish-software.com Mon Apr 16 08:48:07 2018 From: martin at varnish-software.com (Martin Blix Grydeland) Date: Mon, 16 Apr 2018 08:48:07 +0000 Subject: Springtime for VDD ? In-Reply-To: <60611.1523867135@critter.freebsd.dk> References: <58744.1523602806@critter.freebsd.dk> <60611.1523867135@critter.freebsd.dk> Message-ID: I've created a doodle, putting all dates in May that are not public holidays here. https://doodle.com/poll/tbhgzk93p4x9hfx2 Martin On Mon, 16 Apr 2018 at 10:25 Poul-Henning Kamp wrote: > -------- > In message < > CANTn4cq1ayn2imURPL1oBJ4mOh_mnAATuTLoER6fMBFU+GqV6Q at mail.gmail.com>, > Martin Blix Grydeland writes: > > >To get things started, I will suggest Wednesday 9th of May. > > As some of you may have heard, negotiations about public employees > wages are not going too well here in Denmark, and we may have what > amounts to a general strike on our hands pretty soon. Amongst other > things, that will paralyze traffic, so I may not be able to get > anywhere if that happens. > > I still think we should plan on a VDD meeting (see previous > email about a doodle) but it may be wise to not buy unrefundable > flight tickets until we know if I'm stuck in Denmark or not... > > -- > Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 > phk at FreeBSD.ORG | TCP/IP since RFC 956 > FreeBSD committer | BSD since 4.3-tahoe > Never attribute to malice what can adequately be explained by incompetence. > -------------- next part -------------- An HTML attachment was scrubbed... URL: From daghf at varnish-software.com Tue Apr 17 11:32:39 2018 From: daghf at varnish-software.com (Dag Haavi Finstad) Date: Tue, 17 Apr 2018 13:32:39 +0200 Subject: Springtime for VDD ? In-Reply-To: References: <58744.1523602806@critter.freebsd.dk> <60611.1523867135@critter.freebsd.dk> Message-ID: The date has been set: Monday May 14th On Mon, Apr 16, 2018 at 10:48 AM, Martin Blix Grydeland < martin at varnish-software.com> wrote: > I've created a doodle, putting all dates in May that are not public > holidays here. > > https://doodle.com/poll/tbhgzk93p4x9hfx2 > > Martin > > On Mon, 16 Apr 2018 at 10:25 Poul-Henning Kamp wrote: > >> -------- >> In message > gmail.com>, Martin Blix Grydeland writes: >> >> >To get things started, I will suggest Wednesday 9th of May. >> >> As some of you may have heard, negotiations about public employees >> wages are not going too well here in Denmark, and we may have what >> amounts to a general strike on our hands pretty soon. Amongst other >> things, that will paralyze traffic, so I may not be able to get >> anywhere if that happens. >> >> I still think we should plan on a VDD meeting (see previous >> email about a doodle) but it may be wise to not buy unrefundable >> flight tickets until we know if I'm stuck in Denmark or not... >> >> -- >> Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 >> phk at FreeBSD.ORG | TCP/IP since RFC 956 >> FreeBSD committer | BSD since 4.3-tahoe >> Never attribute to malice what can adequately be explained by >> incompetence. >> > > _______________________________________________ > varnish-dev mailing list > varnish-dev at varnish-cache.org > https://www.varnish-cache.org/lists/mailman/listinfo/varnish-dev > -- Dag Haavi Finstad Software Developer | Varnish Software Mobile: +47 476 64 134 We Make Websites Fly! -------------- next part -------------- An HTML attachment was scrubbed... URL: From stephen.butler at gmail.com Fri Apr 27 06:05:56 2018 From: stephen.butler at gmail.com (Stephen J. Butler) Date: Fri, 27 Apr 2018 01:05:56 -0500 Subject: PRIV_CALL, VRT_re_init, and mutexes Message-ID: I'm a new developer/user for Varnish, and was looking to extend a module with some regex support (vmod_cookie if you must know). I don't know anything about vmod development or Varnish threading, so to get an idea of how to do this I looked at the vmod_header module. But I'm confused about why they thought they needed a mutex to protected a call to VRT_re_init. Is this just a useless, historic thing that got left in for some reason? https://github.com/varnish/varnish-modules/blob/master/src/vmod_header.c If you look at vmod_remove it has a 2nd parameter that's in the vcc as PRIV_CALL. This parameter is initialized with the string value from the last parameter (a regex pattern). It does this by calling header_init_re(). And if you look at that function you see: /* * Initialize the regex *s on priv, if it hasn't already been done. * XXX: We have to recheck the condition after grabbing the lock to avoid a * XXX: race condition. */ static void header_init_re(struct vmod_priv *priv, const char *s) { if (priv->priv == NULL) { assert(pthread_mutex_lock(&header_mutex) == 0); if (priv->priv == NULL) { VRT_re_init(&priv->priv, s); priv->free = VRT_re_fini; } pthread_mutex_unlock(&header_mutex); } } My reading of the docs makes me think that this "priv" only lives for each single call of the function. There should be no reason to protect it with a global mutex. Do the internals of VRT_re_init() require this? I can't imagine how, because if it did then this method of protecting it is broken anyway. I suspect this is leftover from some rewrite or updating of the module but wanted to check. -------------- next part -------------- An HTML attachment was scrubbed... URL: From guillaume at varnish-software.com Fri Apr 27 06:37:21 2018 From: guillaume at varnish-software.com (Guillaume Quintard) Date: Fri, 27 Apr 2018 08:37:21 +0200 Subject: PRIV_CALL, VRT_re_init, and mutexes In-Reply-To: References: Message-ID: Hi, and welcome! In this case, priv comes from PRIV_CALL, meaning it's going to be the same for all the request/vcl that call the function from the same spot. This is done to cache the regex, so you only compile for the first call. However, I'm thinking that you may have the same priv with a different string, and then everything goes to hell. Could anyone familiar with PRIV_CALL confirm? Cheers, -- Guillaume Quintard On Fri, Apr 27, 2018 at 8:05 AM, Stephen J. Butler wrote: > I'm a new developer/user for Varnish, and was looking to extend a module > with some regex support (vmod_cookie if you must know). I don't know > anything about vmod development or Varnish threading, so to get an idea of > how to do this I looked at the vmod_header module. > > But I'm confused about why they thought they needed a mutex to protected a > call to VRT_re_init. Is this just a useless, historic thing that got left > in for some reason? > > https://github.com/varnish/varnish-modules/blob/master/src/vmod_header.c > > If you look at vmod_remove it has a 2nd parameter that's in the vcc as > PRIV_CALL. This parameter is initialized with the string value from the > last parameter (a regex pattern). It does this by calling header_init_re(). > And if you look at that function you see: > > /* > * Initialize the regex *s on priv, if it hasn't already been done. > * XXX: We have to recheck the condition after grabbing the lock to avoid a > * XXX: race condition. > */ > static void > header_init_re(struct vmod_priv *priv, const char *s) > { > if (priv->priv == NULL) { > assert(pthread_mutex_lock(&header_mutex) == 0); > if (priv->priv == NULL) { > VRT_re_init(&priv->priv, s); > priv->free = VRT_re_fini; > } > pthread_mutex_unlock(&header_mutex); > } > } > My reading of the docs makes me think that this "priv" only lives for each > single call of the function. There should be no reason to protect it with a > global mutex. Do the internals of VRT_re_init() require this? I can't > imagine how, because if it did then this method of protecting it is broken > anyway. > > I suspect this is leftover from some rewrite or updating of the module but > wanted to check. > > > _______________________________________________ > varnish-dev mailing list > varnish-dev at varnish-cache.org > https://www.varnish-cache.org/lists/mailman/listinfo/varnish-dev > -------------- next part -------------- An HTML attachment was scrubbed... URL: From stephen.butler at gmail.com Fri Apr 27 06:49:05 2018 From: stephen.butler at gmail.com (Stephen J. Butler) Date: Fri, 27 Apr 2018 01:49:05 -0500 Subject: PRIV_CALL, VRT_re_init, and mutexes In-Reply-To: References: Message-ID: Sorry, didn't reply to the list: On Fri, Apr 27, 2018 at 1:48 AM, Stephen J. Butler wrote: > Hmm. If PRIV_CALL is private to the call site, then I think > in vmod_bodyaccess.c, vmod_rematch_req_body()there's a problem. In that > case it calls VRE_compile()to initiate a regex without a lock on a > PRIV_CALL structure. > > On Fri, Apr 27, 2018 at 1:37 AM, Guillaume Quintard < > guillaume at varnish-software.com> wrote: > >> Hi, and welcome! >> >> In this case, priv comes from PRIV_CALL, meaning it's going to be the >> same for all the request/vcl that call the function from the same spot. >> This is done to cache the regex, so you only compile for the first call. >> >> However, I'm thinking that you may have the same priv with a different >> string, and then everything goes to hell. Could anyone familiar with >> PRIV_CALL confirm? >> >> Cheers, >> >> -- >> Guillaume Quintard >> >> On Fri, Apr 27, 2018 at 8:05 AM, Stephen J. Butler < >> stephen.butler at gmail.com> wrote: >> >>> I'm a new developer/user for Varnish, and was looking to extend a module >>> with some regex support (vmod_cookie if you must know). I don't know >>> anything about vmod development or Varnish threading, so to get an idea of >>> how to do this I looked at the vmod_header module. >>> >>> But I'm confused about why they thought they needed a mutex to protected >>> a call to VRT_re_init. Is this just a useless, historic thing that got left >>> in for some reason? >>> >>> https://github.com/varnish/varnish-modules/blob/master/src/vmod_header.c >>> >>> If you look at vmod_remove it has a 2nd parameter that's in the vcc as >>> PRIV_CALL. This parameter is initialized with the string value from the >>> last parameter (a regex pattern). It does this by calling >>> header_init_re(). And if you look at that function you see: >>> >>> /* >>> * Initialize the regex *s on priv, if it hasn't already been done. >>> * XXX: We have to recheck the condition after grabbing the lock to avoid >>> a >>> * XXX: race condition. >>> */ >>> static void >>> header_init_re(struct vmod_priv *priv, const char *s) >>> { >>> if (priv->priv == NULL) { >>> assert(pthread_mutex_lock(&header_mutex) == 0); >>> if (priv->priv == NULL) { >>> VRT_re_init(&priv->priv, s); >>> priv->free = VRT_re_fini; >>> } >>> pthread_mutex_unlock(&header_mutex); >>> } >>> } >>> My reading of the docs makes me think that this "priv" only lives for >>> each single call of the function. There should be no reason to protect it >>> with a global mutex. Do the internals of VRT_re_init() require this? I >>> can't imagine how, because if it did then this method of protecting it is >>> broken anyway. >>> >>> I suspect this is leftover from some rewrite or updating of the module >>> but wanted to check. >>> >>> >>> _______________________________________________ >>> varnish-dev mailing list >>> varnish-dev at varnish-cache.org >>> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-dev >>> >> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From stephen.butler at gmail.com Fri Apr 27 07:05:28 2018 From: stephen.butler at gmail.com (Stephen J. Butler) Date: Fri, 27 Apr 2018 02:05:28 -0500 Subject: PRIV_CALL, VRT_re_init, and mutexes In-Reply-To: References: Message-ID: In the varnish source code for the 6.0 release, I found one use of PRIV_CALL in vmod_std, vmod_fileread(). This function does have locks, but they're all to protect modifications of the global frlist variable. It does seem like the priv parameter is used as a call site value but no care is taken to make sure access is exclusive. In that case it also bothers me that there's an assumption that -- and ++ are atomic operations on all platforms and compilers... some quick Googling suggests that's not the case. On Fri, Apr 27, 2018 at 1:49 AM, Stephen J. Butler wrote: > Sorry, didn't reply to the list: > > On Fri, Apr 27, 2018 at 1:48 AM, Stephen J. Butler < > stephen.butler at gmail.com> wrote: > >> Hmm. If PRIV_CALL is private to the call site, then I think >> in vmod_bodyaccess.c, vmod_rematch_req_body()there's a problem. In that >> case it calls VRE_compile()to initiate a regex without a lock on a >> PRIV_CALL structure. >> >> On Fri, Apr 27, 2018 at 1:37 AM, Guillaume Quintard < >> guillaume at varnish-software.com> wrote: >> >>> Hi, and welcome! >>> >>> In this case, priv comes from PRIV_CALL, meaning it's going to be the >>> same for all the request/vcl that call the function from the same spot. >>> This is done to cache the regex, so you only compile for the first call. >>> >>> However, I'm thinking that you may have the same priv with a different >>> string, and then everything goes to hell. Could anyone familiar with >>> PRIV_CALL confirm? >>> >>> Cheers, >>> >>> -- >>> Guillaume Quintard >>> >>> On Fri, Apr 27, 2018 at 8:05 AM, Stephen J. Butler < >>> stephen.butler at gmail.com> wrote: >>> >>>> I'm a new developer/user for Varnish, and was looking to extend a >>>> module with some regex support (vmod_cookie if you must know). I don't know >>>> anything about vmod development or Varnish threading, so to get an idea of >>>> how to do this I looked at the vmod_header module. >>>> >>>> But I'm confused about why they thought they needed a mutex to >>>> protected a call to VRT_re_init. Is this just a useless, historic thing >>>> that got left in for some reason? >>>> >>>> https://github.com/varnish/varnish-modules/blob/master/src/v >>>> mod_header.c >>>> >>>> If you look at vmod_remove it has a 2nd parameter that's in the vcc as >>>> PRIV_CALL. This parameter is initialized with the string value from the >>>> last parameter (a regex pattern). It does this by calling >>>> header_init_re(). And if you look at that function you see: >>>> >>>> /* >>>> * Initialize the regex *s on priv, if it hasn't already been done. >>>> * XXX: We have to recheck the condition after grabbing the lock to >>>> avoid a >>>> * XXX: race condition. >>>> */ >>>> static void >>>> header_init_re(struct vmod_priv *priv, const char *s) >>>> { >>>> if (priv->priv == NULL) { >>>> assert(pthread_mutex_lock(&header_mutex) == 0); >>>> if (priv->priv == NULL) { >>>> VRT_re_init(&priv->priv, s); >>>> priv->free = VRT_re_fini; >>>> } >>>> pthread_mutex_unlock(&header_mutex); >>>> } >>>> } >>>> My reading of the docs makes me think that this "priv" only lives for >>>> each single call of the function. There should be no reason to protect it >>>> with a global mutex. Do the internals of VRT_re_init() require this? I >>>> can't imagine how, because if it did then this method of protecting it is >>>> broken anyway. >>>> >>>> I suspect this is leftover from some rewrite or updating of the module >>>> but wanted to check. >>>> >>>> >>>> _______________________________________________ >>>> varnish-dev mailing list >>>> varnish-dev at varnish-cache.org >>>> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-dev >>>> >>> >>> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From stephen.butler at gmail.com Fri Apr 27 07:06:46 2018 From: stephen.butler at gmail.com (Stephen J. Butler) Date: Fri, 27 Apr 2018 02:06:46 -0500 Subject: PRIV_CALL, VRT_re_init, and mutexes In-Reply-To: References: Message-ID: (sorry, never mind about the -- and ++; they're all inside the lock) On Fri, Apr 27, 2018 at 2:05 AM, Stephen J. Butler wrote: > In the varnish source code for the 6.0 release, I found one use of > PRIV_CALL in vmod_std, vmod_fileread(). This function does have locks, but > they're all to protect modifications of the global frlist variable. It does > seem like the priv parameter is used as a call site value but no care is > taken to make sure access is exclusive. > > In that case it also bothers me that there's an assumption that -- and ++ > are atomic operations on all platforms and compilers... some quick Googling > suggests that's not the case. > > > On Fri, Apr 27, 2018 at 1:49 AM, Stephen J. Butler < > stephen.butler at gmail.com> wrote: > >> Sorry, didn't reply to the list: >> >> On Fri, Apr 27, 2018 at 1:48 AM, Stephen J. Butler < >> stephen.butler at gmail.com> wrote: >> >>> Hmm. If PRIV_CALL is private to the call site, then I think >>> in vmod_bodyaccess.c, vmod_rematch_req_body()there's a problem. In that >>> case it calls VRE_compile()to initiate a regex without a lock on a >>> PRIV_CALL structure. >>> >>> On Fri, Apr 27, 2018 at 1:37 AM, Guillaume Quintard < >>> guillaume at varnish-software.com> wrote: >>> >>>> Hi, and welcome! >>>> >>>> In this case, priv comes from PRIV_CALL, meaning it's going to be the >>>> same for all the request/vcl that call the function from the same spot. >>>> This is done to cache the regex, so you only compile for the first call. >>>> >>>> However, I'm thinking that you may have the same priv with a different >>>> string, and then everything goes to hell. Could anyone familiar with >>>> PRIV_CALL confirm? >>>> >>>> Cheers, >>>> >>>> -- >>>> Guillaume Quintard >>>> >>>> On Fri, Apr 27, 2018 at 8:05 AM, Stephen J. Butler < >>>> stephen.butler at gmail.com> wrote: >>>> >>>>> I'm a new developer/user for Varnish, and was looking to extend a >>>>> module with some regex support (vmod_cookie if you must know). I don't know >>>>> anything about vmod development or Varnish threading, so to get an idea of >>>>> how to do this I looked at the vmod_header module. >>>>> >>>>> But I'm confused about why they thought they needed a mutex to >>>>> protected a call to VRT_re_init. Is this just a useless, historic thing >>>>> that got left in for some reason? >>>>> >>>>> https://github.com/varnish/varnish-modules/blob/master/src/v >>>>> mod_header.c >>>>> >>>>> If you look at vmod_remove it has a 2nd parameter that's in the vcc as >>>>> PRIV_CALL. This parameter is initialized with the string value from the >>>>> last parameter (a regex pattern). It does this by calling >>>>> header_init_re(). And if you look at that function you see: >>>>> >>>>> /* >>>>> * Initialize the regex *s on priv, if it hasn't already been done. >>>>> * XXX: We have to recheck the condition after grabbing the lock to >>>>> avoid a >>>>> * XXX: race condition. >>>>> */ >>>>> static void >>>>> header_init_re(struct vmod_priv *priv, const char *s) >>>>> { >>>>> if (priv->priv == NULL) { >>>>> assert(pthread_mutex_lock(&header_mutex) == 0); >>>>> if (priv->priv == NULL) { >>>>> VRT_re_init(&priv->priv, s); >>>>> priv->free = VRT_re_fini; >>>>> } >>>>> pthread_mutex_unlock(&header_mutex); >>>>> } >>>>> } >>>>> My reading of the docs makes me think that this "priv" only lives for >>>>> each single call of the function. There should be no reason to protect it >>>>> with a global mutex. Do the internals of VRT_re_init() require this? I >>>>> can't imagine how, because if it did then this method of protecting it is >>>>> broken anyway. >>>>> >>>>> I suspect this is leftover from some rewrite or updating of the module >>>>> but wanted to check. >>>>> >>>>> >>>>> _______________________________________________ >>>>> varnish-dev mailing list >>>>> varnish-dev at varnish-cache.org >>>>> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-dev >>>>> >>>> >>>> >>> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From stephen.butler at gmail.com Fri Apr 27 07:47:22 2018 From: stephen.butler at gmail.com (Stephen J. Butler) Date: Fri, 27 Apr 2018 02:47:22 -0500 Subject: PRIV_CALL, VRT_re_init, and mutexes In-Reply-To: References: Message-ID: OK, I think I figured it out. PRIV_CALL is a call site specific storage. I ended up figuring this out by looking at the compiled VCL output from varnishd. For headers there should probably be documentation to not use a variable argument for the pattern. In vmod_bodyaccess the same thing (no variables for patterns) but it should also lock around the VRE_compile, and the free function is incorrect and should be VRE_free, or it leaks memory during race conditions. On Fri, Apr 27, 2018 at 2:06 AM, Stephen J. Butler wrote: > (sorry, never mind about the -- and ++; they're all inside the lock) > > On Fri, Apr 27, 2018 at 2:05 AM, Stephen J. Butler < > stephen.butler at gmail.com> wrote: > >> In the varnish source code for the 6.0 release, I found one use of >> PRIV_CALL in vmod_std, vmod_fileread(). This function does have locks, but >> they're all to protect modifications of the global frlist variable. It does >> seem like the priv parameter is used as a call site value but no care is >> taken to make sure access is exclusive. >> >> In that case it also bothers me that there's an assumption that -- and ++ >> are atomic operations on all platforms and compilers... some quick Googling >> suggests that's not the case. >> >> >> On Fri, Apr 27, 2018 at 1:49 AM, Stephen J. Butler < >> stephen.butler at gmail.com> wrote: >> >>> Sorry, didn't reply to the list: >>> >>> On Fri, Apr 27, 2018 at 1:48 AM, Stephen J. Butler < >>> stephen.butler at gmail.com> wrote: >>> >>>> Hmm. If PRIV_CALL is private to the call site, then I think >>>> in vmod_bodyaccess.c, vmod_rematch_req_body()there's a problem. In >>>> that case it calls VRE_compile()to initiate a regex without a lock on >>>> a PRIV_CALL structure. >>>> >>>> On Fri, Apr 27, 2018 at 1:37 AM, Guillaume Quintard < >>>> guillaume at varnish-software.com> wrote: >>>> >>>>> Hi, and welcome! >>>>> >>>>> In this case, priv comes from PRIV_CALL, meaning it's going to be the >>>>> same for all the request/vcl that call the function from the same spot. >>>>> This is done to cache the regex, so you only compile for the first call. >>>>> >>>>> However, I'm thinking that you may have the same priv with a different >>>>> string, and then everything goes to hell. Could anyone familiar with >>>>> PRIV_CALL confirm? >>>>> >>>>> Cheers, >>>>> >>>>> -- >>>>> Guillaume Quintard >>>>> >>>>> On Fri, Apr 27, 2018 at 8:05 AM, Stephen J. Butler < >>>>> stephen.butler at gmail.com> wrote: >>>>> >>>>>> I'm a new developer/user for Varnish, and was looking to extend a >>>>>> module with some regex support (vmod_cookie if you must know). I don't know >>>>>> anything about vmod development or Varnish threading, so to get an idea of >>>>>> how to do this I looked at the vmod_header module. >>>>>> >>>>>> But I'm confused about why they thought they needed a mutex to >>>>>> protected a call to VRT_re_init. Is this just a useless, historic thing >>>>>> that got left in for some reason? >>>>>> >>>>>> https://github.com/varnish/varnish-modules/blob/master/src/v >>>>>> mod_header.c >>>>>> >>>>>> If you look at vmod_remove it has a 2nd parameter that's in the vcc >>>>>> as PRIV_CALL. This parameter is initialized with the string value from the >>>>>> last parameter (a regex pattern). It does this by calling >>>>>> header_init_re(). And if you look at that function you see: >>>>>> >>>>>> /* >>>>>> * Initialize the regex *s on priv, if it hasn't already been done. >>>>>> * XXX: We have to recheck the condition after grabbing the lock to >>>>>> avoid a >>>>>> * XXX: race condition. >>>>>> */ >>>>>> static void >>>>>> header_init_re(struct vmod_priv *priv, const char *s) >>>>>> { >>>>>> if (priv->priv == NULL) { >>>>>> assert(pthread_mutex_lock(&header_mutex) == 0); >>>>>> if (priv->priv == NULL) { >>>>>> VRT_re_init(&priv->priv, s); >>>>>> priv->free = VRT_re_fini; >>>>>> } >>>>>> pthread_mutex_unlock(&header_mutex); >>>>>> } >>>>>> } >>>>>> My reading of the docs makes me think that this "priv" only lives for >>>>>> each single call of the function. There should be no reason to protect it >>>>>> with a global mutex. Do the internals of VRT_re_init() require this? I >>>>>> can't imagine how, because if it did then this method of protecting it is >>>>>> broken anyway. >>>>>> >>>>>> I suspect this is leftover from some rewrite or updating of the >>>>>> module but wanted to check. >>>>>> >>>>>> >>>>>> _______________________________________________ >>>>>> varnish-dev mailing list >>>>>> varnish-dev at varnish-cache.org >>>>>> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-dev >>>>>> >>>>> >>>>> >>>> >>> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: