From slink at schokola.de Wed May 2 14:50:22 2018 From: slink at schokola.de (Nils Goroll) Date: Wed, 2 May 2018 16:50:22 +0200 Subject: backend/director admin states Message-ID: this is a follow up to a discussion between phk and myself on #varnish-hacking today: phk is in the process of restructuring director/backend APIs. One of the areas currently reconsidered is the admin health status. As of now (68a78f94b3a4e702edf73dea4e11e626c4c20301), the admin health has become private to varnishd (off limits to vmods) and semantics have changed. We discussed which admin health states should exist and their semantics. Notice that for the "probe" admin health, the actual health state is determined by the probe. Here's my proposal: * auto this state is the default and only exists as an argument to varnishadm backend.set_health. It resolves to either - "probe" if the backend/director has a probe - "dependent" if the director has no probe, but a healthy callback - "healthy" otherwise (no probe and no healthy callback) * healthy The backend/director is considered unconditionally healthy. Connections will be attempted. This state is used for backends without a probe and can be manually set using varnishadm backend.set_health and from vmods. * sick The backend/director is considered unconditionally sick. Connections will not be attempted. Can be manually set using varnishadm backend.set_health and from vmods. * disabled The health state is determined as for "probe" or "healthy" but a newly added VRT / vmod_std function (e.g. backend_disabled()) returns true. Directors may offer the choice to use disabled backends or not. Can be manually set using varnishadm backend.set_health and from vmods. The use case is to flag a backend for plannend maintenance such that it is used for existing sessions, but not for new ones. * probe The health state is determined by a probe, yet connections can still be attempted even if the probe is unsuccessful and directors may offer the choice to still hand out unhealthy backends. This state can not be set explicitly, but is selected through varnishadm backend.set_health auto or from vmods. * dependent The health state of the director is determined by the backends it refers to. This state can not be set explicitly, but is selected through varnishadm backend.set_health auto or from vmods. * deleted The backend has been deleted (via a vmod) will be removed once all references have been cleared. Illustrated example: varnishadm backend.list output Backend name Admin Probe Last change boot.b1 probe 0/8 bad Wed, 02 May 2018 13:22:10 GMT boot.b2 probe 8/8 good Wed, 02 May 2018 13:22:10 GMT boot.b3 healthy 1/8 bad Wed, 02 May 2018 13:22:10 GMT boot.b4 healthy - Wed, 02 May 2018 13:22:10 GMT boot.b5 sick 7/8 good Wed, 02 May 2018 13:22:10 GMT boot.b6 sick - Wed, 02 May 2018 13:22:10 GMT boot.b7 disabled 7/8 good Wed, 02 May 2018 13:22:10 GMT boot.b8 deleted - Wed, 02 May 2018 13:22:10 GMT boot.d1 dependent - Wed, 02 May 2018 13:22:10 GMT boot.d2 probe 3/4 good Wed, 02 May 2018 13:22:10 GMT in detail: * boot.b1 probe 0/8 bad backend with a probe for which all recent health checks have failed result from set_health=auto or default * boot.b2 probe 8/8 good backend with a probe for which all recent health checks have succeeded result from set_health=auto or default * boot.b3 healthy 1/8 bad backend with a probe for which most recent health checks have failed result from set_health=healthy * boot.b4 healthy - backend with no probe result from set_health=healthy, set_health=auto or default * boot.b5 sick 7/8 good backend with a probe for which most recent health checks have succeeded result from set_health=sick * boot.b6 sick - backend with no probe result from set_health=sick * boot.b7 disabled 7/8 good backend with a probe for which most recent health checks have succeeded result from set_health=disabled custom vcl logic will not use this backend for new sessions * boot.b8 deleted - backend has been removed (by a vmod) * boot.d1 dependent - director without a probe result from set_health=auto or default * boot.d2 probe 3/4 good director with a probe, which returns 3/4 success (may be the number of healthy backends) result from set_health=auto or default From guillaume at varnish-software.com Wed May 2 15:07:24 2018 From: guillaume at varnish-software.com (Guillaume Quintard) Date: Wed, 2 May 2018 17:07:24 +0200 Subject: backend/director admin states In-Reply-To: References: Message-ID: I think I could live with "auto" meaning "dependent/probe" depending on the type of object. Not sure the disabled case is super useful. If I get what you say, the only thing differing from setting it to sick is that we keep using the open connections, instead of finishing in-flight requests. Is it worth it? I'd argue "deleted" should be "dead", just to complete the medically depressing vocabulary list you started :-) (and we'll need to bring back "saint" from the dead) -- Guillaume Quintard On Wed, May 2, 2018 at 4:50 PM, Nils Goroll wrote: > this is a follow up to a discussion between phk and myself on > #varnish-hacking > today: > > phk is in the process of restructuring director/backend APIs. One of the > areas > currently reconsidered is the admin health status. As of now > (68a78f94b3a4e702edf73dea4e11e626c4c20301), the admin health has become > private > to varnishd (off limits to vmods) and semantics have changed. > > We discussed which admin health states should exist and their semantics. > Notice > that for the "probe" admin health, the actual health state is determined > by the > probe. > > Here's my proposal: > > * auto > > this state is the default and only exists as an argument to varnishadm > backend.set_health. It resolves to either > > - "probe" if the backend/director has a probe > - "dependent" if the director has no probe, but a healthy callback > - "healthy" otherwise (no probe and no healthy callback) > > * healthy > > The backend/director is considered unconditionally healthy. Connections > will > be attempted. > > This state is used for backends without a probe and can be manually set > using varnishadm backend.set_health and from vmods. > > * sick > > The backend/director is considered unconditionally sick. Connections > will not > be attempted. > > Can be manually set using varnishadm backend.set_health and from vmods. > > * disabled > > The health state is determined as for "probe" or "healthy" but a newly > added > VRT / vmod_std function (e.g. backend_disabled()) returns true. > Directors may > offer the choice to use disabled backends or not. > > Can be manually set using varnishadm backend.set_health and from vmods. > > The use case is to flag a backend for plannend maintenance such that it > is > used for existing sessions, but not for new ones. > > * probe > > The health state is determined by a probe, yet connections can still be > attempted even if the probe is unsuccessful and directors may offer the > choice > to still hand out unhealthy backends. > > This state can not be set explicitly, but is selected through > varnishadm backend.set_health auto or from vmods. > > * dependent > > The health state of the director is determined by the backends it refers > to. > > This state can not be set explicitly, but is selected through > varnishadm backend.set_health auto or from vmods. > > * deleted > > The backend has been deleted (via a vmod) will be removed once all > references > have been cleared. > > > > Illustrated example: varnishadm backend.list output > > Backend name Admin Probe Last change > boot.b1 probe 0/8 bad Wed, 02 May 2018 > 13:22:10 GMT > boot.b2 probe 8/8 good Wed, 02 May 2018 > 13:22:10 GMT > boot.b3 healthy 1/8 bad Wed, 02 May 2018 > 13:22:10 GMT > boot.b4 healthy - Wed, 02 May 2018 > 13:22:10 GMT > boot.b5 sick 7/8 good Wed, 02 May 2018 > 13:22:10 GMT > boot.b6 sick - Wed, 02 May 2018 > 13:22:10 GMT > boot.b7 disabled 7/8 good Wed, 02 May 2018 > 13:22:10 GMT > boot.b8 deleted - Wed, 02 May 2018 > 13:22:10 GMT > boot.d1 dependent - Wed, 02 May 2018 > 13:22:10 GMT > boot.d2 probe 3/4 good Wed, 02 May 2018 > 13:22:10 GMT > > in detail: > > * boot.b1 probe 0/8 bad > > backend with a probe for which all recent health checks have failed > > result from set_health=auto or default > > * boot.b2 probe 8/8 good > > backend with a probe for which all recent health checks have succeeded > > result from set_health=auto or default > > * boot.b3 healthy 1/8 bad > > backend with a probe for which most recent health checks have failed > > result from set_health=healthy > > * boot.b4 healthy - > > backend with no probe > > result from set_health=healthy, set_health=auto or default > > * boot.b5 sick 7/8 good > > backend with a probe for which most recent health checks have succeeded > > result from set_health=sick > > * boot.b6 sick - > > backend with no probe > > result from set_health=sick > > * boot.b7 disabled 7/8 good > > backend with a probe for which most recent health checks have succeeded > > result from set_health=disabled > > custom vcl logic will not use this backend for new sessions > > * boot.b8 deleted - > > backend has been removed (by a vmod) > > * boot.d1 dependent - > > director without a probe > > result from set_health=auto or default > > * boot.d2 probe 3/4 good > > director with a probe, which returns 3/4 success (may be the number of > healthy > backends) > > result from set_health=auto or default > _______________________________________________ > varnish-dev mailing list > varnish-dev at varnish-cache.org > https://www.varnish-cache.org/lists/mailman/listinfo/varnish-dev > -------------- next part -------------- An HTML attachment was scrubbed... URL: From slink at schokola.de Wed May 2 15:24:11 2018 From: slink at schokola.de (Nils Goroll) Date: Wed, 2 May 2018 17:24:11 +0200 Subject: backend/director admin states In-Reply-To: References: Message-ID: <29bd4410-3533-3753-ea2f-a25a21213438@schokola.de> Hi, On 02/05/18 17:07, Guillaume Quintard wrote: > I think I could live with "auto" meaning "dependent/probe" depending on the type > of object. Me too, but I also think that for varnishadm backend.list output, displaying the actual facts is significantly more helpful. > Not sure the disabled case is super useful. If I get what you say, the only > thing differing from setting it to sick is that we keep using the open > connections, instead of finishing in-flight requests. Is it worth it? No, the suggestion is regarding a different level: Consider some application server backends with local session store: Using shard / hash / explicit vcl, you have implemented some kind of persistence such that requests with the same session id always get sent to the same backend. Now you plan to restart that backend and want to still send requests with existing persisted session ids to it, but no requests without a session cookie. One current way to do this is to essentially define every backend twice and use once director for the "existing session" and one for the "no session" case. Other options probably involve std.healthy(), the backend probe and retries. I think we can do much better than this. > I'd argue "deleted" should be "dead", just to complete the medically depressing > vocabulary list you started :-) wfm. > (and we'll need to bring back "saint" from the dead) Greetings from the guy who currently tries to keep the saint's mode working with. ;) Nils From dridi at varni.sh Thu May 3 08:49:23 2018 From: dridi at varni.sh (Dridi Boukelmoune) Date: Thu, 3 May 2018 10:49:23 +0200 Subject: backend/director admin states In-Reply-To: References: Message-ID: > Illustrated example: varnishadm backend.list output > > Backend name Admin Probe Last change > boot.b1 probe 0/8 bad Wed, 02 May 2018 13:22:10 GMT > boot.b2 probe 8/8 good Wed, 02 May 2018 13:22:10 GMT > boot.b3 healthy 1/8 bad Wed, 02 May 2018 13:22:10 GMT > boot.b4 healthy - Wed, 02 May 2018 13:22:10 GMT > boot.b5 sick 7/8 good Wed, 02 May 2018 13:22:10 GMT > boot.b6 sick - Wed, 02 May 2018 13:22:10 GMT > boot.b7 disabled 7/8 good Wed, 02 May 2018 13:22:10 GMT > boot.b8 deleted - Wed, 02 May 2018 13:22:10 GMT > boot.d1 dependent - Wed, 02 May 2018 13:22:10 GMT > boot.d2 probe 3/4 good Wed, 02 May 2018 13:22:10 GMT We should make this output a bit more functional if we're going to break the output. Columns: - name - type (std, roundrobin, fallback, ...) - health (auto healthy or sick as in backend.set_health, and deleted/dead) - probe (vcl name of the probe for std backends, arbitrary string for directors) - state (depends on the director, for example "bad" or "good") - comment (eg. 0/8 for native probes) - last change And as usual "-" when there's no value. Example: > Name Type Health Probe State Comment Last change > boot.www1 std auto none good - Wed, 02 May 2018 13:22:10 GMT > boot.www2 std sick none bad - Wed, 02 May 2018 13:22:10 GMT > boot.www fallback auto cluster good - Wed, 02 May 2018 13:22:10 GMT The idea is to avoid blanks inside columns (except "last change/updated") to make it more friendly to a ny column oriented toollike /bin/sh's `read` or awk's fields. Directors can then contribute to backend.list -p by pretty-printing what can explain their current state (eg. list of backends for our built-in directors). Dridi From geoff at uplex.de Thu May 3 09:06:23 2018 From: geoff at uplex.de (Geoff Simmons) Date: Thu, 3 May 2018 11:06:23 +0200 Subject: backend/director admin states In-Reply-To: References: Message-ID: <1401d370-56dd-6006-2b3d-21771650ff48@uplex.de> On 05/03/2018 10:49 AM, Dridi Boukelmoune wrote: >> Illustrated example: varnishadm backend.list output >> >> Backend name Admin Probe Last change >> boot.b1 probe 0/8 bad Wed, 02 May 2018 13:22:10 GMT >> boot.b2 probe 8/8 good Wed, 02 May 2018 13:22:10 GMT >> boot.b3 healthy 1/8 bad Wed, 02 May 2018 13:22:10 GMT >> boot.b4 healthy - Wed, 02 May 2018 13:22:10 GMT >> boot.b5 sick 7/8 good Wed, 02 May 2018 13:22:10 GMT >> boot.b6 sick - Wed, 02 May 2018 13:22:10 GMT >> boot.b7 disabled 7/8 good Wed, 02 May 2018 13:22:10 GMT >> boot.b8 deleted - Wed, 02 May 2018 13:22:10 GMT >> boot.d1 dependent - Wed, 02 May 2018 13:22:10 GMT >> boot.d2 probe 3/4 good Wed, 02 May 2018 13:22:10 GMT > > We should make this output a bit more functional if we're going to > break the output. [...] > Example: > >> Name Type Health Probe State Comment Last change >> boot.www1 std auto none good - Wed, 02 May 2018 13:22:10 GMT >> boot.www2 std sick none bad - Wed, 02 May 2018 13:22:10 GMT >> boot.www fallback auto cluster good - Wed, 02 May 2018 13:22:10 GMT > > The idea is to avoid blanks inside columns (except "last change/updated") > to make it more friendly to a ny column oriented toollike /bin/sh's `read` or > awk's fields. I would also suggest ISO 8601 format for the 'Last change' column, to make it more easily parseable: 2018-05-02T13:22:10Z Then every whitespace-separated field corresponds to a data column. Best, Geoff -- ** * * UPLEX - Nils Goroll Systemoptimierung Scheffelstra?e 32 22301 Hamburg Tel +49 40 2880 5731 Mob +49 176 636 90917 Fax +49 40 42949753 http://uplex.de -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 833 bytes Desc: OpenPGP digital signature URL: From phk at phk.freebsd.dk Thu May 3 09:20:45 2018 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Thu, 03 May 2018 09:20:45 +0000 Subject: backend/director admin states In-Reply-To: <1401d370-56dd-6006-2b3d-21771650ff48@uplex.de> References: <1401d370-56dd-6006-2b3d-21771650ff48@uplex.de> Message-ID: <71112.1525339245@critter.freebsd.dk> OK, let me try to explain my mental model of this: Admin health is owned by the CLI and director implementations do not get to meddle with it. (But they do get to see it) Admin=sick means sick, no matter what the director implementation thinks. Admin=healthy means healthy, --//-- Admin=??? means "Implementation decides if sick or healthy." Best proposal for "???" seems to be "auto", even though it is not entirely on point. The implementation can control its view of the health two ways A) Call VRT_SetHealth() and set it on the director. (useful for probes as we know them) B) Provide a ->healthy() method which will be asked whenever anybody wants to know. (useful for selecting directors like rr/fb/rnd/hash) In the new API, there is a director method called ->list() which gets called during CLI command 'backend.list' so that UX can reflect what the implementation actually know as best as possible. So as I see it: We need to decide: 1) Best word for "???" above. 2) Headers in backend.list 3) What does H1 backend without probe do for ->list() 4) What does H1 backend with probe do for ->list() 5) What do vmod_directors do for ->list() Poul-Henning PS: Related stuff: I'm pondering a "VCL_BACKEND std.resolve_backend(VCL_BACKEND)" to avoid what seems to me to be a lot of complexity in shard to do the same? -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From dridi at varni.sh Thu May 3 10:00:03 2018 From: dridi at varni.sh (Dridi Boukelmoune) Date: Thu, 3 May 2018 12:00:03 +0200 Subject: backend/director admin states In-Reply-To: <1401d370-56dd-6006-2b3d-21771650ff48@uplex.de> References: <1401d370-56dd-6006-2b3d-21771650ff48@uplex.de> Message-ID: > I would also suggest ISO 8601 format for the 'Last change' column, to > make it more easily parseable: > > 2018-05-02T13:22:10Z > > Then every whitespace-separated field corresponds to a data column. Very good point, and this format is a natural fit for sorting! Dridi From dridi at varni.sh Thu May 3 10:02:53 2018 From: dridi at varni.sh (Dridi Boukelmoune) Date: Thu, 3 May 2018 12:02:53 +0200 Subject: backend/director admin states In-Reply-To: <71112.1525339245@critter.freebsd.dk> References: <1401d370-56dd-6006-2b3d-21771650ff48@uplex.de> <71112.1525339245@critter.freebsd.dk> Message-ID: > Best proposal for "???" seems to be "auto", even though it is not > entirely on point. > > The implementation can control its view of the health two ways > > A) Call VRT_SetHealth() and set it on the director. > (useful for probes as we know them) > > B) Provide a ->healthy() method which will be asked whenever > anybody wants to know. > (useful for selecting directors like rr/fb/rnd/hash) "auto" is what we have today in the CLI, and I don't remember ever being confused by that when I was only a user. And it seems to still match your A) and B) cases. Dridi From phk at phk.freebsd.dk Thu May 3 10:52:01 2018 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Thu, 03 May 2018 10:52:01 +0000 Subject: backend/director admin states In-Reply-To: References: <1401d370-56dd-6006-2b3d-21771650ff48@uplex.de> <71112.1525339245@critter.freebsd.dk> Message-ID: <72229.1525344721@critter.freebsd.dk> -------- In message , Dridi Boukelmoune writes: >> Best proposal for "???" seems to be "auto", even though it is not >> entirely on point. >> >> The implementation can control its view of the health two ways >> >> A) Call VRT_SetHealth() and set it on the director. >> (useful for probes as we know them) >> >> B) Provide a ->healthy() method which will be asked whenever >> anybody wants to know. >> (useful for selecting directors like rr/fb/rnd/hash) > >"auto" is what we have today in the CLI, and I don't remember ever >being confused by that when I was only a user. And it seems to >still match your A) and B) cases. Where it doesn't match is a backend without probe/other mechanism. -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From phk at phk.freebsd.dk Thu May 3 10:52:50 2018 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Thu, 03 May 2018 10:52:50 +0000 Subject: backend/director admin states In-Reply-To: <1401d370-56dd-6006-2b3d-21771650ff48@uplex.de> References: <1401d370-56dd-6006-2b3d-21771650ff48@uplex.de> Message-ID: <72242.1525344770@critter.freebsd.dk> -------- In message <1401d370-56dd-6006-2b3d-21771650ff48 at uplex.de>, Geoff Simmons write s: >>> Illustrated example: varnishadm backend.list output We should also think about varnishadm backend.list -j ? -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From guillaume at varnish-software.com Thu May 3 10:56:41 2018 From: guillaume at varnish-software.com (Guillaume Quintard) Date: Thu, 03 May 2018 10:56:41 +0000 Subject: backend/director admin states In-Reply-To: <72242.1525344770@critter.freebsd.dk> References: <1401d370-56dd-6006-2b3d-21771650ff48@uplex.de> <72242.1525344770@critter.freebsd.dk> Message-ID: All commands should have a JSON option, I reckon. -- Guillaume Quintard On Thu, May 3, 2018, 12:52 Poul-Henning Kamp wrote: > -------- > In message <1401d370-56dd-6006-2b3d-21771650ff48 at uplex.de>, Geoff Simmons > write > s: > > >>> Illustrated example: varnishadm backend.list output > > We should also think about varnishadm backend.list -j ? > > -- > Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 > phk at FreeBSD.ORG | TCP/IP since RFC 956 > FreeBSD committer | BSD since 4.3-tahoe > Never attribute to malice what can adequately be explained by incompetence. > _______________________________________________ > varnish-dev mailing list > varnish-dev at varnish-cache.org > https://www.varnish-cache.org/lists/mailman/listinfo/varnish-dev > -------------- next part -------------- An HTML attachment was scrubbed... URL: From phk at phk.freebsd.dk Thu May 3 11:04:29 2018 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Thu, 03 May 2018 11:04:29 +0000 Subject: backend/director admin states In-Reply-To: References: <1401d370-56dd-6006-2b3d-21771650ff48@uplex.de> <72242.1525344770@critter.freebsd.dk> Message-ID: <72398.1525345469@critter.freebsd.dk> -------- In message , Guillaume Quintard writes: >All commands should have a JSON option, I reckon. Somebody should write more code for it then, I reckon :-) -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From guillaume at varnish-software.com Thu May 3 11:23:55 2018 From: guillaume at varnish-software.com (Guillaume Quintard) Date: Thu, 03 May 2018 11:23:55 +0000 Subject: backend/director admin states In-Reply-To: <72398.1525345469@critter.freebsd.dk> References: <1401d370-56dd-6006-2b3d-21771650ff48@uplex.de> <72242.1525344770@critter.freebsd.dk> <72398.1525345469@critter.freebsd.dk> Message-ID: Yuuuup, that's on my plate. Which is rather large... -- Guillaume Quintard On Thu, May 3, 2018, 13:04 Poul-Henning Kamp wrote: > -------- > In message c5Q-ZxFqJCU0kPEw at mail.gmail.com> > , Guillaume Quintard writes: > > >All commands should have a JSON option, I reckon. > > Somebody should write more code for it then, I reckon :-) > > -- > Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 > phk at FreeBSD.ORG | TCP/IP since RFC 956 > FreeBSD committer | BSD since 4.3-tahoe > Never attribute to malice what can adequately be explained by incompetence. > -------------- next part -------------- An HTML attachment was scrubbed... URL: From dridi at varni.sh Thu May 3 11:51:35 2018 From: dridi at varni.sh (Dridi Boukelmoune) Date: Thu, 3 May 2018 13:51:35 +0200 Subject: backend/director admin states In-Reply-To: <72398.1525345469@critter.freebsd.dk> References: <1401d370-56dd-6006-2b3d-21771650ff48@uplex.de> <72242.1525344770@critter.freebsd.dk> <72398.1525345469@critter.freebsd.dk> Message-ID: On Thu, May 3, 2018 at 1:04 PM, Poul-Henning Kamp wrote: > -------- > In message > , Guillaume Quintard writes: > >>All commands should have a JSON option, I reckon. > > Somebody should write more code for it then, I reckon :-) Maybe we should do that the other way around? Have all CLI commands reply with a JSON data structure, and CLI consumers like varnishadm, varnishtest or varnishd -d be able to output it in plain text (with this code centralized in libvarnish and exposed in libvarnishapi). Dridi From phk at phk.freebsd.dk Fri May 4 06:53:40 2018 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Fri, 04 May 2018 06:53:40 +0000 Subject: backend/director admin states In-Reply-To: References: <1401d370-56dd-6006-2b3d-21771650ff48@uplex.de> <72242.1525344770@critter.freebsd.dk> <72398.1525345469@critter.freebsd.dk> Message-ID: <76937.1525416820@critter.freebsd.dk> -------- In message , Dridi Boukelmoune writes: >On Thu, May 3, 2018 at 1:04 PM, Poul-Henning Kamp wrote: >> -------- >> In message >> , Guillaume Quintard writes: >> >>>All commands should have a JSON option, I reckon. >> >> Somebody should write more code for it then, I reckon :-) > >Maybe we should do that the other way around? Have all CLI commands >reply with a JSON data structure, and CLI consumers like varnishadm, >varnishtest or varnishd -d be able to output it in plain text (with >this code centralized in libvarnish and exposed in libvarnishapi). I thought about that, but didn't move on it, as I didn't want to write all the code myself ;-) -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From dridi at varni.sh Fri May 4 07:24:42 2018 From: dridi at varni.sh (Dridi Boukelmoune) Date: Fri, 4 May 2018 09:24:42 +0200 Subject: backend/director admin states In-Reply-To: <76937.1525416820@critter.freebsd.dk> References: <1401d370-56dd-6006-2b3d-21771650ff48@uplex.de> <72242.1525344770@critter.freebsd.dk> <72398.1525345469@critter.freebsd.dk> <76937.1525416820@critter.freebsd.dk> Message-ID: >>Maybe we should do that the other way around? Have all CLI commands >>reply with a JSON data structure, and CLI consumers like varnishadm, >>varnishtest or varnishd -d be able to output it in plain text (with >>this code centralized in libvarnish and exposed in libvarnishapi). > > I thought about that, but didn't move on it, as I didn't want to > write all the code myself ;-) I experimented with this a while back and tried this: - have a CLI commands reply with a JSON payload - same as today: fixed-length size, newline, payload - includes metadata such as the response status - JSON payload includes the "plain text" renderer's name - example: "backend.list", "backend.list -p" etc - one per command plus variants that change the output format - didn't go as far as plain text rendering - replace the response JSON payload with custom byte code - the render chooses between JSON or text rendering - byte code is JSON-like, but easier to parse - one generic JSON renderer for all commands - started experimenting with plain text conversion - code generation from a schema for each output type - example: "backend.list", "backend.list -p" etc (same as before) - defensive assertions that let us know when we break the output - the idea was to start actually versioning the CLI from 2.0 on - and at least bump y in 2.y when we add stuff to the CLI I never completed the last experiment because the time I allocated eventually ran out and I lost motivation to nail down the schema part. This was in the varnishreload [1] time frame, when I realized that changes to the CLI broke the reload scripts we ship in the [4.1,5.x] series and since we were maintaining them out of tree we had no easy way to keep track of that. Which is why I also wanted to import varnishreload upstream and have proper coverage. I have thrown the code away, but the idea stuck. And it was a fun experiment :) Dridi [1] https://github.com/varnishcache/pkg-varnish-cache/pull/70 From phk at phk.freebsd.dk Fri May 4 07:42:33 2018 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Fri, 04 May 2018 07:42:33 +0000 Subject: backend/director admin states In-Reply-To: References: <1401d370-56dd-6006-2b3d-21771650ff48@uplex.de> <72242.1525344770@critter.freebsd.dk> <72398.1525345469@critter.freebsd.dk> <76937.1525416820@critter.freebsd.dk> Message-ID: <77145.1525419753@critter.freebsd.dk> -------- In message , Dridi Boukelmoune writes: >I experimented with this a while back and tried this: > >[...] > >I never completed the last experiment because the time I allocated >eventually ran out and I lost motivation to nail down the schema part. I appreciate the effort, having personally been down that road a number of times in my career, all in vain. I think my conclusion at this point is that I have not seen an actual schema work anywhere - ever. That is not to say that schemas cannot ever work, they proabably can in certain limited circumstances. The reason we have include/tbl/cli_cmds.h in the first place was that I wanted some kind of "schema-light" for UX use, but that didn't happen. Instead varnishadm issues a "help" command, to get the _actual_ commands available. Varnishadm was a convenience, and the intent was that people could connect to the CLI with any random script they cared to write. Once we did the PSK-authentication, telnet'ing became a lot less relevant, and varnishadm became the defacto CLI access tool. Therefore it is relevant to consider a design-change so varnishd only responds with JSON (but it can and should be nicely formatted JSON), and varnishadm gets to render that nicely for the user. Amongst many other UX advantages, varnishadm can know the width of terminal windows, and the availability of colors and exploit that. But if you try to impose a schema between varnishd and varnishadm, you either loose access to all the really nifty things varnishadm could do, or you spend the rest of your life trying to define a schema which can express them. I would argue that almost everybody have given up on schemas by now, and switched to shipping code: Your average website sends a shitload of JS code and datastructures for it to act on to your browser. I don't see a need for varnishd to ship code to varnishadm though, so I'd be happy with a design where varnishd CLI always spits out JSON, and varnishadm is responsible for formatting it, if required. As long as somebody who is not me writes the code for it... PS: We have a solid JSON parser in the tree already. -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From dridi at varni.sh Fri May 4 08:00:45 2018 From: dridi at varni.sh (Dridi Boukelmoune) Date: Fri, 4 May 2018 10:00:45 +0200 Subject: backend/director admin states In-Reply-To: <77145.1525419753@critter.freebsd.dk> References: <1401d370-56dd-6006-2b3d-21771650ff48@uplex.de> <72242.1525344770@critter.freebsd.dk> <72398.1525345469@critter.freebsd.dk> <76937.1525416820@critter.freebsd.dk> <77145.1525419753@critter.freebsd.dk> Message-ID: On Fri, May 4, 2018 at 9:42 AM, Poul-Henning Kamp wrote: > -------- > The reason we have include/tbl/cli_cmds.h in the first place was > that I wanted some kind of "schema-light" for UX use, but that > didn't happen. Yes, that experiment failed, but I somehow succeeded in doing something similar for varnishtest: a schema for commands and a generic parser dispatches TheRightThing(tm) to callbacks. It worked with nested commands too although I only implemented a dozen in total. The generic parser squared some of the concurrency issues too by keeping track of all parties. I don't have time these days for such experiments :( > I don't see a need for varnishd to ship code to varnishadm though, > so I'd be happy with a design where varnishd CLI always spits out > JSON, and varnishadm is responsible for formatting it, if required. The reason I wanted to centralize the json2txt code is to also ensure that varnishd -d and varnishtest logs stay readable. If we want varnishadm to take advantage of its interactive mode to align columns and whatnots, we can always have a json2matrix step and then render it. > As long as somebody who is not me writes the code for it... > > PS: We have a solid JSON parser in the tree already. All we need is TheNewGuy(tm) to dump this on their desk :p Cheers From phk at phk.freebsd.dk Fri May 4 08:16:24 2018 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Fri, 04 May 2018 08:16:24 +0000 Subject: backend/director admin states In-Reply-To: References: <1401d370-56dd-6006-2b3d-21771650ff48@uplex.de> <72242.1525344770@critter.freebsd.dk> <72398.1525345469@critter.freebsd.dk> <76937.1525416820@critter.freebsd.dk> <77145.1525419753@critter.freebsd.dk> Message-ID: <67574.1525421784@critter.freebsd.dk> -------- In message , Dridi Boukelmoune writes: >All we need is TheNewGuy(tm) to dump this on their desk :p Indeed, they grow up so fast, don't they ? :-) Anyway, I would like to get back to the actual underlying state-machinery for directors... -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From dridi at varni.sh Mon May 7 08:08:19 2018 From: dridi at varni.sh (Dridi Boukelmoune) Date: Mon, 7 May 2018 10:08:19 +0200 Subject: [VDD] VSL query extensions Message-ID: Hello everyone, I'd like to not put this item on the agenda for next Monday so I'll briefly present it here since nobody at Varnish Software thinks it will be controversial. Attendees outside of Varnish Software, please ack that this should have a de-facto "send patches" green light or we can discuss this further next Monday. It comes in 3 simple steps. 1) Multi-line queries When someone wishes to capture logs for multiple reasons, they need to OR them, possibly with brackets. Instead the proposal is to support one query per line, ignore empty lines, and treat tokens starting with '#' as comments. Example: varnishlog -q '(my-first-query) or (my-second-query)' varnishlog -q ' my-first-query my-second-query ... ' 2) Read queries from a file Similar to reading varnishncsa's format from a file: cat >bad-things.txt <= 500 # backend problems ... # internal incident 4253 ... # varnish issue 1799 ... EOF varnishlog -Q bad-things.txt 3) Query scoping The current scope is always the transaction group, but sometimes we may want conditions to apply to a subset of the `-g` option (for example two fields of the same VSL record). Proposed solution: inside brackets have the possibility to reduce the scope with the very first token. The scope cannot outlive the parent scope, and the query keeps its default @group transaction. Example: # beresp cached more than 2m before v_b_r{} @tag TTL[1] eq RFC and TTL[2] >= 120 The groups would be: - @group (default when omitted, matches -g option) - @request (same as -g request) - @vxid (same as -g vxid) - @tag (single record) Things that shouldn't work: varnishlog -q '@request ...' varnishlog -q '@tag ... and (@group ...)' varnishlog -g vxid -q '@request ...' varnishlog -q '@tag SomeRecord and DifferentRecord' My opinion was that the current VSL query architecture should accommodate such changes nicely, and Martin is also not too worried about it. 4) Bonus topic I also wanted to put a small performance topic on the agenda: mmap'ing regular files when reading offline VSL. The current code treats files (-r option) the same way live SHMLOG VSL is and allocates copies. The idea is to mmap regular files and avoid needless copies, and keep the current `-r file` handling for the stdin case. I hope to speed up processing when it comes to dumps in the GBs. Someone asked about binary VSL future plans. All of this should be orthogonal to changes in the underlying VSL format. Nothing should require an soname major bump in libvarnishapi (minor bump for the -Q option). Dridi From dridi at varni.sh Mon May 7 08:22:28 2018 From: dridi at varni.sh (Dridi Boukelmoune) Date: Mon, 7 May 2018 10:22:28 +0200 Subject: [VDD] Client-side latency Message-ID: Hello everyone, I'd like to not put this item on the agenda for next Monday so I'll briefly present it here since nobody at Varnish Software thinks it will be controversial. Attendees outside of Varnish Software, please ack that this should have a de-facto "send patches" green light or we can discuss this further next Monday. I have recently run into simple latency problems that were tricky to observe in Varnish. First, I ran into a case where we didn't have enough workers and sessions were queued. The client would report response times that wouldn't match what we observed in varnishlog because Timestamp record don't account for queuing (and it becomes tricky with h2). Second, we then realized that the client was also too eager on pipelining and many requests would pile up a "long" time before Varnish would process them. Once again everything looks good in varnishlog. Proposed feature: A new %{Varnish:date}x variable in varnishncsa that converts req.http.date or beresp.http.date into a Timestamp-like format. This should help spot "significant" differences between what the peer claims and what the Timestamp:Start or Timestamp:Beresp records say. Dridi From slink at schokola.de Wed May 9 09:34:02 2018 From: slink at schokola.de (Nils Goroll) Date: Wed, 9 May 2018 11:34:02 +0200 Subject: backend/director admin states / std.resolve_backend? / PRIV_REQ? In-Reply-To: <71112.1525339245@critter.freebsd.dk> References: <1401d370-56dd-6006-2b3d-21771650ff48@uplex.de> <71112.1525339245@critter.freebsd.dk> Message-ID: I guess this thread still lacks my response to phks reply: On 03/05/18 11:20, Poul-Henning Kamp wrote: > Admin health is owned by the CLI and director implementations do not > get to meddle with it. (But they do get to see it) I understand the point, but allowing vmods to change the admin health would make sense in the same way as the ban() cli command makes sense: in some cases, controlling such functionality in-band avoids additional complications. > Admin=sick means sick, no matter what the director implementation thinks. > > Admin=healthy means healthy, --//-- > > Admin=??? means "Implementation decides if sick or healthy." > > Best proposal for "???" seems to be "auto", even though it is not > entirely on point. Ack. But I'd still like to see "auto" be a write-only health state, resulting in healthy or whatever the implementation returns for ->list(). > In the new API, there is a director method called ->list() which > gets called during CLI command 'backend.list' so that UX can reflect > what the implementation actually know as best as possible. It seems to make a lot of sense to replace "probe" and "dependent" from my proposal with an implementation specific value. So ->list() would fill out both the fields replacing the current "Admin" and "Probe" fields. > 2) Headers in backend.list Admin and Check ? > 3) What does H1 backend without probe do for ->list() return Admin=Healthy Check=- > 4) What does H1 backend with probe do for ->list() return Admin=Healthy Check= > 5) What do vmod_directors do for ->list() sensible default: return Admin=dependent Check=n/m number of healthy/total backends in the director Also I'd like to push again for the "disabled" admin state: Yes, this could be implemented with another director layer, but I think the additional flag belongs to the fundamental director because the proposed functionality is fundamental and can be found in all the loadbalancer appliances I've encountered so far. > I'm pondering a "VCL_BACKEND std.resolve_backend(VCL_BACKEND)" to > avoid what seems to me to be a lot of complexity in shard to do > the same? The complexity in shard is not for resolve=NOW, but for resolve=LAZY: We need to keep track of all the load balancing parameters without actually making the decision. At this point, they are stored in a PRIV_TASK which cannot cross the client/backend boundary, so we need to special case based on which side we're running on. I think that replacing resolve=NOW with std.resolve_backend() would make the situation even more confusing for users as resolve=LAZY has limitations and resolve=NOW doesn't. If anything, with the current code, std.resolve_backend() would make sense for the the directors EXCEPT for shard. If we wanted to generalize more, I think we'd need something like a PRIV_REQ: - can be created on the client side - ownership transferred to the backend side or fini'd if no backend request Nils From slink at schokola.de Wed May 9 10:08:13 2018 From: slink at schokola.de (Nils Goroll) Date: Wed, 9 May 2018 12:08:13 +0200 Subject: backend/director admin states / std.resolve_backend? / PRIV_REQ? In-Reply-To: References: <1401d370-56dd-6006-2b3d-21771650ff48@uplex.de> <71112.1525339245@critter.freebsd.dk> Message-ID: On 09/05/18 11:34, Nils Goroll wrote: > ban() cli VCL that is From slink at schokola.de Wed May 9 15:28:48 2018 From: slink at schokola.de (Nils Goroll) Date: Wed, 9 May 2018 17:28:48 +0200 Subject: VDI_Resolve -> VRT, default behaviour for no healthy() cb Message-ID: I'm picking up two loose threads from #varnish-hacking: VDI_Resolve ----------- I'm working on a director which needs to resolve another (layered) director argument, so I'd prefer to have basically VDI_Resolve in VRT. Proposed signature: VCL_BACKEND VRT_Resolve(VRT_CTX, VCL_BACKEND) the VCL_BACKEND argument would be used instead of bo->director_req. Also I'd want to move the logging calls to the caller. avoid director healthy() callback --------------------------------- To avoid code duplication, I think core code should basically call VRT_Healthy(VRT_Resolve(ctx, backend)); if there is no healthy callback. IMHO this would make much more sense than just asserting a director without a healthy callback to be healthy always. Thx, Nils From slink at schokola.de Wed May 9 15:53:00 2018 From: slink at schokola.de (Nils Goroll) Date: Wed, 9 May 2018 17:53:00 +0200 Subject: VDI_Resolve -> VRT, default behaviour for no healthy() cb In-Reply-To: References: Message-ID: On 09/05/18 17:28, Nils Goroll wrote: > VDI_Resolve > ----------- > > I'm working on a director which needs to resolve another (layered) director > argument for those interested, I've pushed a working prototype to a public repo: https://code.uplex.de/uplex-varnish/libvmod-cluster From slink at schokola.de Mon May 14 15:42:35 2018 From: slink at schokola.de (Nils Goroll) Date: Mon, 14 May 2018 17:42:35 +0200 Subject: save the date hamburg vdd Message-ID: <7a1372d8-5998-335d-6025-f1fb91628a7e@schokola.de> 27.9. optional VDD if we want 28.9. VDD 29.9. optional fun day including TMBG concert *) or techno party depending on taste *) need to book yourself if you want to go From phk at phk.freebsd.dk Tue May 15 06:24:38 2018 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Tue, 15 May 2018 06:24:38 +0000 Subject: VDD summary Message-ID: <3198.1526365478@critter.freebsd.dk> Thanks to everybody at the VDD in Oslo yesterday, I think we got a lot done and we certainly ended up with one of the most actionable summaries we have ever had. See you in Hamburg next time: 27.9. optional VDD if we want 28.9. VDD 29.9. optional fun day including TMBG concert *) or techno party depending on taste *) need to book yourself if you want to go H2::Send_Frame move to ->error #1799 Pluggable ops for TCP-Pools and TCP operations => Proper way to do this is to generalize and come up with multi-protocol backend support => Unknown/large scope: unlikely for a Sept release #2573 @^#@&^#@-Headers (vmod_header[s]) => New vmod_header. @Steven writes/proposes .vcc file => New parameter: header-charset ? return 400 if not acceptable. Default=what VCL can handle. SELinux: maintain a policy upstream => we ship a policy if we can test it (dridi to figure this out) (That really goes for *anything* in our tree) CLI:JSON: => First step is implement -j in all commands VFP/VDP in vmods => If filter-list is set, setting do_{gzip|gunzip|esi} gives Failure libvgz compilation warnings inflate.c:1232:25: error: this statement may fall through [-Werror=implicit-fallthrough=] state->mode = LENGTH; ~~~~~~~~~~~~^~~~~~~~ either: -Wno-error=implicit-fallthrough= if the code supports it, or just -Wno-error (new zlib code is still affected by this) Reported upstream: https://github.com/madler/zlib/issues/316 => separate "varnish" CFLAGS (warnings) from the rest => don't pass varnish CFLAGS to foreign code => don't treat sanitizer flags separately? Features for Sept 2018 release? VCL variables? (see https://etherpad.wikimedia.org/p/vdd17q3 lines 103-114) release proc/docs etc. * docs: changes.rst / vs. sphinx/whats-new/changes-X.X.rst new definition: doc/changes.rst only as a hint to release managers #1799 => P?l wrote this: https://github.com/varnishcache/varnish-cache/compare/master...hermunn:master-cache-behavior-vmods and Martin has added some comments. Also, Nils had some comments in an email. => Implement func pointer zegerman_catflap() veto function from HSH_Lookup{} slink to write PR => Reimplement req.grace, vcl_hit{} becomes "return(deliver)". (P?l does a PR for this) Director/Refcount/Probe => phk suggests BACKEND std.resolve(BACKEND) (slink says it doesn't work for shard directors, which phk thinks is shard's problem, not his.) -> explained and would be solved by cookie idea below -> a general mechanism for passing private data from the client to backend side may be interesting --> describe some use cases to phk to persuade him of the usefulness -> this may be where introducing variables becomes interesting, e.g. declare INT req.var.foo, which then becomes available as an INT as bereq.var.foo XXX: why wouldn't it simply by var.foo both places ? => new CLI command: backend.tell args => ... if glob matches multiple backends, error-returns are ignored => ... Should "?" args be magic ? => "lightweight" directors for req.backend_hint, storing "cookie" on req.ws which is copied to bereq.ws and reinstated. (?) => What happens if backend_hint is shard with shards under it ? How do 2nd-level shards get cookies ? -> slink volunteers to write PR if phk wants him to VDP cleanup and API changes (#2673) => Sounds fine, consider slink's comments, reiterate, then PHK to review VRT_CTX, priv->free(), calls outside VCL methods (VDI) (and object destructors?) => This one sounds sensible, but requires some homework wrt. breaking APIs and if priv's need broken more (ie: dynamic objects) (again) native VCC type for regexen => VCL_REGEXP, compiler turns CONST-STRING into vre Next VDD on Norway-Denmark boat ? No: TMBG concert in Hamburg -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From dridi at varni.sh Tue May 15 14:36:09 2018 From: dridi at varni.sh (Dridi Boukelmoune) Date: Tue, 15 May 2018 16:36:09 +0200 Subject: VDD summary In-Reply-To: <3198.1526365478@critter.freebsd.dk> References: <3198.1526365478@critter.freebsd.dk> Message-ID: > => new CLI command: backend.tell args > => ... if glob matches multiple backends, error-returns are ignored > => ... Should "?" args be magic ? It occurred to me that directors listening to the CLI via a backend.tell command is a very special case that we could otherwise generalize to a vmod.cmd CLI command. This would allow modules like xkey to purge either from VCL or the CLI, just like ban: vmod.cmd command [args...] Example: vmod.cmd xkey help help [] purge [-s] [...] vmod.cmd xkey help purge purge [-s] [...] Perform a purge of all objects matching at least one id, or a soft purge when the -s flag is present. vmod.cmd xkey purge 123 42 objects purged. This way it's up to directors to have their respective VMODs expose various ways to change their state. Thanks for the VDD and thanks to Nils for his patience and explanations of their backend/director advanced needs and how we can keep the user interface manageable in VCL. Dridi From phk at phk.freebsd.dk Wed May 16 07:20:53 2018 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Wed, 16 May 2018 07:20:53 +0000 Subject: VDD summary In-Reply-To: References: <3198.1526365478@critter.freebsd.dk> Message-ID: <8792.1526455253@critter.freebsd.dk> -------- In message , Dridi Boukelmoune writes: >> => new CLI command: backend.tell args >> => ... if glob matches multiple backends, error-returns are ignored >> => ... Should "?" args be magic ? > >It occurred to me that directors listening to the CLI via a backend.tell >command is a very special case that we could otherwise generalize >to a vmod.cmd CLI command. This would allow modules like xkey >to purge either from VCL or the CLI, just like ban: > > vmod.cmd command [args...] That's a sensible generalization, but in difference from the backend targeted version, it will be up to the vmod itself to look a string up to find a backend etc. -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From dridi at varni.sh Wed May 16 07:51:47 2018 From: dridi at varni.sh (Dridi Boukelmoune) Date: Wed, 16 May 2018 09:51:47 +0200 Subject: VDD summary In-Reply-To: <8792.1526455253@critter.freebsd.dk> References: <3198.1526365478@critter.freebsd.dk> <8792.1526455253@critter.freebsd.dk> Message-ID: On Wed, May 16, 2018 at 9:20 AM, Poul-Henning Kamp wrote: > -------- > In message > , Dridi Boukelmoune writes: >>> => new CLI command: backend.tell args >>> => ... if glob matches multiple backends, error-returns are ignored >>> => ... Should "?" args be magic ? >> >>It occurred to me that directors listening to the CLI via a backend.tell >>command is a very special case that we could otherwise generalize >>to a vmod.cmd CLI command. This would allow modules like xkey >>to purge either from VCL or the CLI, just like ban: >> >> vmod.cmd command [args...] > > That's a sensible generalization, but in difference from the > backend targeted version, it will be up to the vmod itself to > look a string up to find a backend etc. And you may have several competing vmods loaded with the same name thanks to vmod caching. I'm aware of the can of worms it opens (competing vmods, temperature, active vs labels...) but thought it would be worth mentioning. Then it's up to the director implementation to come batteries^Wcli-included and upstream Varnish could set an example with vmod_directors while Varnish Software could probably find a couple cases like the one I outlined for vmod_xkey to help the generalization. I think that many people would be happy to issue xkey purges via the CLI much like bans and not have to worry about ACLs or other security concerns. Cheers From phk at phk.freebsd.dk Wed May 16 07:56:04 2018 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Wed, 16 May 2018 07:56:04 +0000 Subject: VDD summary In-Reply-To: References: <3198.1526365478@critter.freebsd.dk> <8792.1526455253@critter.freebsd.dk> Message-ID: <9029.1526457364@critter.freebsd.dk> -------- In message , Dridi Boukelmoune writes: >And you may have several competing vmods loaded with the same name >thanks to vmod caching. Well, due to having multiple loaded VCLs really. It would have to be possible to say vmod.tell somevcl::somevmod "bla bla bla" To target a single one, but it is probably ill-advised to insist on always only targeting a single one. -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From geoff at uplex.de Wed May 16 08:34:35 2018 From: geoff at uplex.de (Geoff Simmons) Date: Wed, 16 May 2018 10:34:35 +0200 Subject: VDD summary In-Reply-To: <9029.1526457364@critter.freebsd.dk> References: <3198.1526365478@critter.freebsd.dk> <8792.1526455253@critter.freebsd.dk> <9029.1526457364@critter.freebsd.dk> Message-ID: On 05/16/2018 09:56 AM, Poul-Henning Kamp wrote: > > It would have to be possible to say > > vmod.tell somevcl::somevmod "bla bla bla" > > To target a single one, but it is probably ill-advised to insist > on always only targeting a single one. ... and then it's easy to imagine syntax like: # Mumble to somevmod in every loaded instance vmod.tell *::somevmod "mumble" # Mumble to somevmod in the active instance vmod.tell $$ACTIVE::somevmod "mumble" Assuming that some string $$ACTIVE can be formed that may not be the name of any VCL instance. I doubt that we need somevcl::* or *::*, since it's unlikely that there's a command that every VMOD's CLI will understand. Unless for some reason we think of a command that any VMOD with a CLI should support, which I guess is not out of the question after all. Think of something like "give me back a version string for the VMOD". Best, Geoff -- ** * * UPLEX - Nils Goroll Systemoptimierung Scheffelstra?e 32 22301 Hamburg Tel +49 40 2880 5731 Mob +49 176 636 90917 Fax +49 40 42949753 http://uplex.de -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 833 bytes Desc: OpenPGP digital signature URL: From phk at phk.freebsd.dk Fri May 18 09:30:16 2018 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Fri, 18 May 2018 09:30:16 +0000 Subject: CLI command vmod.tell In-Reply-To: References: <3198.1526365478@critter.freebsd.dk> <8792.1526455253@critter.freebsd.dk> <9029.1526457364@critter.freebsd.dk> Message-ID: <3821.1526635816@critter.freebsd.dk> -------- In message , Geoff Simmons write s: >> vmod.tell somevcl::somevmod "bla bla bla" I looked a bit at this. There is a hurt of locking issues for VMOD writers trying to do anything smart, but trivial stuff like setting global options for the VMOD etc, shouldn't cause trouble. VCC wise I think this will be a "$CLI" stanza, and the function will get called with a VRT_CTX and a char **, and return an enum VCLI_status_e. I think I prefer this naming: vmod.tell [VCL_PATTERN:]VMODNAME arguments [...] The default VCL_PATTERN will be the active VCL. In other words: vmod.tell std bugger off only hits the std vmod in the active vcl, whereas vmod.tell *:std bugger off will hit the std vmod in all vcls which has imported it. When used without a pattern: ---------------------------- If the active VCL has not imported a vmod of that name status is CLIS_CANT. If that VMOD does not implement $CLI it will be CLIS_UNIMPL. Otherwise the VMODs $CLI determines the status. When used with a pattern: ------------------------- If no VCLs are matched by pattern: CLIS_CANT If the VMOD is not loaded in any of the matched VCLs: CLIS_CANT If no hit VMOD implements $CLI: CLIS_UNIMPL. If any hit VMOD's $CLI returns non-200: new status CLIS_KABOOM (=202) Otherwise CLIS_OK Still waffling on: ------------------ Should names of VCLs be put into cli output as subheadings when pattern is used so VMODs won't have to do that, or should they only be added in front of VMODs which produce cli output. -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From dridi at varni.sh Fri May 18 14:04:20 2018 From: dridi at varni.sh (Dridi Boukelmoune) Date: Fri, 18 May 2018 16:04:20 +0200 Subject: CLI command vmod.tell In-Reply-To: <3821.1526635816@critter.freebsd.dk> References: <3198.1526365478@critter.freebsd.dk> <8792.1526455253@critter.freebsd.dk> <9029.1526457364@critter.freebsd.dk> <3821.1526635816@critter.freebsd.dk> Message-ID: On Fri, May 18, 2018 at 11:30 AM, Poul-Henning Kamp wrote: > -------- > In message , Geoff Simmons write > s: > >>> vmod.tell somevcl::somevmod "bla bla bla" > > I looked a bit at this. > > There is a hurt of locking issues for VMOD writers trying to do > anything smart, but trivial stuff like setting global options for > the VMOD etc, shouldn't cause trouble. > > VCC wise I think this will be a "$CLI" stanza, and the function > will get called with a VRT_CTX and a char **, and return an enum > VCLI_status_e. Can we also pass a nonce to the VMOD? So that "global" actions can be skipped once done when a VMOD like xkey is targeted with a VCL pattern matching more than once. > I think I prefer this naming: > > vmod.tell [VCL_PATTERN:]VMODNAME arguments [...] > > The default VCL_PATTERN will be the active VCL. > > In other words: > > vmod.tell std bugger off > > only hits the std vmod in the active vcl, whereas > > vmod.tell *:std bugger off > > will hit the std vmod in all vcls which has imported it. > > When used without a pattern: > ---------------------------- > > If the active VCL has not imported a vmod of that name > status is CLIS_CANT. > > If that VMOD does not implement $CLI it will be CLIS_UNIMPL. > > Otherwise the VMODs $CLI determines the status. > > When used with a pattern: > ------------------------- > > If no VCLs are matched by pattern: CLIS_CANT > > If the VMOD is not loaded in any of the matched VCLs: CLIS_CANT > > If no hit VMOD implements $CLI: CLIS_UNIMPL. > > If any hit VMOD's $CLI returns non-200: new status CLIS_KABOOM (=202) > > Otherwise CLIS_OK > > Still waffling on: > ------------------ > > Should names of VCLs be put into cli output as subheadings when > pattern is used so VMODs won't have to do that, or should they > only be added in front of VMODs which produce cli output. > > -- > Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 > phk at FreeBSD.ORG | TCP/IP since RFC 956 > FreeBSD committer | BSD since 4.3-tahoe > Never attribute to malice what can adequately be explained by incompetence. > _______________________________________________ > varnish-dev mailing list > varnish-dev at varnish-cache.org > https://www.varnish-cache.org/lists/mailman/listinfo/varnish-dev From phk at phk.freebsd.dk Fri May 18 14:07:58 2018 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Fri, 18 May 2018 14:07:58 +0000 Subject: CLI command vmod.tell In-Reply-To: References: <3198.1526365478@critter.freebsd.dk> <8792.1526455253@critter.freebsd.dk> <9029.1526457364@critter.freebsd.dk> <3821.1526635816@critter.freebsd.dk> Message-ID: <4671.1526652478@critter.freebsd.dk> -------- In message , Dridi Boukelmoune writes: >> vmod.tell [VCL_PATTERN:]VMODNAME arguments [...] >Can we also pass a nonce to the VMOD? So that "global" actions can be >skipped once done when a VMOD like xkey is targeted with a VCL pattern >matching more than once. You are entirely in control of what "arguments" are and how they are interpreted. -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From dridi at varni.sh Fri May 18 14:15:00 2018 From: dridi at varni.sh (Dridi Boukelmoune) Date: Fri, 18 May 2018 16:15:00 +0200 Subject: CLI command vmod.tell In-Reply-To: <4671.1526652478@critter.freebsd.dk> References: <3198.1526365478@critter.freebsd.dk> <8792.1526455253@critter.freebsd.dk> <9029.1526457364@critter.freebsd.dk> <3821.1526635816@critter.freebsd.dk> <4671.1526652478@critter.freebsd.dk> Message-ID: On Fri, May 18, 2018 at 4:07 PM, Poul-Henning Kamp wrote: > -------- > In message > , Dridi Boukelmoune writes: > >>> vmod.tell [VCL_PATTERN:]VMODNAME arguments [...] > >>Can we also pass a nonce to the VMOD? So that "global" actions can be >>skipped once done when a VMOD like xkey is targeted with a VCL pattern >>matching more than once. > > You are entirely in control of what "arguments" are and how > they are interpreted. I mean an argument generated by varnish, not something I would entrust to the CLI user. Dridi