From scan-admin at coverity.com  Mon Jun  2 14:25:31 2025
From: scan-admin at coverity.com (scan-admin at coverity.com)
Date: Mon, 02 Jun 2025 14:25:31 +0000 (UTC)
Subject: Coverity Scan: Analysis completed for varnish
Message-ID: <683db45a5069f_1439262b604e8ef998578cc@prd-scan-dashboard-0.mail>

An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/varnish-dev/attachments/20250602/2b6bffc7/attachment.html>

From scan-admin at coverity.com  Mon Jun  9 12:40:29 2025
From: scan-admin at coverity.com (scan-admin at coverity.com)
Date: Mon, 09 Jun 2025 12:40:29 +0000 (UTC)
Subject: Coverity Scan: Analysis completed for varnish
Message-ID: <6846d63d5c494_1b3a942b604e8ef998578f9@prd-scan-dashboard-0.mail>

An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/varnish-dev/attachments/20250609/e36e3d05/attachment.html>

From scan-admin at coverity.com  Mon Jun 16 09:13:21 2025
From: scan-admin at coverity.com (scan-admin at coverity.com)
Date: Mon, 16 Jun 2025 09:13:21 +0000 (UTC)
Subject: Coverity Scan: Analysis completed for varnish
Message-ID: <684fe03168ed2_511f42c80f4a439a814353@prd-scan-dashboard-0.mail>

An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/varnish-dev/attachments/20250616/ed6c68ed/attachment.html>

From scan-admin at coverity.com  Mon Jun 23 11:58:48 2025
From: scan-admin at coverity.com (scan-admin at coverity.com)
Date: Mon, 23 Jun 2025 11:58:48 +0000 (UTC)
Subject: Coverity Scan: Analysis completed for varnish
Message-ID: <6859417787c3a_c1ee22c80f4a439a814363@prd-scan-dashboard-0.mail>

An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/varnish-dev/attachments/20250623/b71db2d5/attachment-0001.html>

From slink at schokola.de  Wed Jun 25 08:18:32 2025
From: slink at schokola.de (Nils Goroll)
Date: Wed, 25 Jun 2025 10:18:32 +0200
Subject: Fleshed out ideas from VDD25Q2
In-Reply-To: <202505280653.54S6rEaf022850@critter.freebsd.dk>
References: <202505280653.54S6rEaf022850@critter.freebsd.dk>
Message-ID: <26e1fbc3-a7bb-4a07-a9ba-7ac1a0108879@schokola.de>

Sorry for the long delay...

On 28.05.25 08:53, Poul-Henning Kamp wrote:
> A.1) Make it possible to import and export backends(=directors) between VCLs.

I think this is a good idea, which does not throw too many previous design 
decisions overboard, yet still achieves "global-ish" director semantics.

> I wont go into the details (compatibility with the vcl_method they
> are called from), but that runs into an equally old dogma:
>
> 	"If you can vcl.load a VCL, you can vcl.use that VCL."

Do we want to introduce VCL types? Should the "backend + acl VCL" be of an "aux" 
type such that vcl.use can fail and VCC knows to only support certain VCL objects?

> This originated in a desire to have a preloaded, ever-ready "emergency
> VCL" so that when the newspaper backend monster keeled over, there
> would be a single /reliable/ switch to throw.

What is the new aspect of this? What is missing today? I would think one would 
just "vcl.load emergency <file>" and call "vcl.use emergency" when needed?

> So for now: I think we should implement export/import of backends
> and ACLs, since I think they "come for free", but not commit
> to sharing SUBs.

+1, but see below

> A.1) Thoughts about implementation
> [...]
> For the same reasons as for return(vcl), the imports have to go
> through labels, otherwise "the other" VCL cannot be replaced.

Besides the naming being an unpainted bike shed (import and include are taken), 
I would like to elaborate on how this would work:

a) change backend definitions

So let's say we have in main.vcl:

	use directors * from "directors";

Then "directors" would be a label. To update the directors, one would change the 
"directors" label to point to a different vcl. But in this case, what happens 
with reference checking? If main.vcl uses backend b, but the new vcl which the 
"directors" label now points to no longer defines it, do we notice this when 
changing the label or how else?

b) VMOD-defined directors

In my real world, most backends are dynamic. So to be useful in any capacity, 
the "aux vcl" would at least need to also support VMOD defined directors as in

	sub vcl_init {
		new dyn = dynamic.director(...);
		export dyn;
	}

which the "main vcl" would then use as

	use object dyn from "directors";

	sub vcl_backend_fetch {
		set bereq.backend = dyn.backend(...);
	}

Because VMOD defined directors are just VCL objects, this would generalize 
nicely, I think, but we would also sneak in VCL subs by means of SUB references 
which can be handed to VCL object constructors... That said, SUB references have 
context checking at runtime, so we avoid issues which otherwise exist with 
compile time checking.

> A.2) Add a central switchboard.
> 
> I think the final version of the idea we came up with, was
> something like this mock-up:
> 
> 	vcl 4.2;
> 
> 	vcl_scope {
> 		req.http.host suffix "example.com";
> 		req.url prefix "/hello_world";
> 		return(mine(100));
> 	}
> 
> These "selectors" will be merged into a single decision data structure
> which a central dispatcher uses to decide where the requests goes.

I am of the *strong* opinion that we should *not* do this and stay away from any 
kind of "select the vcl by bult-in logic X", as has been discussed for a bit 
during the VDD. In my mind, we have label switching and should improve it (by 
allowing recursive use of labels). This (already today) allows to use efficient 
"classifier" constructs written in VCL to make a decision *1). We should _not_ 
move this decision away from explicit VCL code.

WHY? Because any "baked in" logic we will come up with for vcl_scope {} will 
have limits on what you can express, and in that case one will have to revert to 
a "switchboard.vcl" with explicit (return(label)) anyway.

Also, I am going to play phk here, anyone suggesting this should come up with a 
strawman for how to resolve conflicts in the definitions. I have seen this kind 
of thing being done wrong at least by a specific service of a popular CDN named 
a* and a popular scale out service named k*, so this problem has a history of 
being under-estimated.

What *would* make more sense, IMHO, was to support this use case with a bundled 
templating tool. This could generate the "switchboard.vcl" from something 
similar to vcl_scope {}. In the form of magic comments, this can be done today.

The generated "switchboard.vcl" can then either be used directly, or included 
from customized vcl, which is free to override it in any way possible. And why 
not copy the semantics of the built-in vcl?


> I think we also had consensus for adding an escape mechanism along
> the lines of:
> 
> 	vcl 4.2;
> 
> 	sub vcl_match {
> 		if (client.ip ~ inhouse_acl && req.url ~ "editor") {
> 			return (mine(100));
> 		}
> 		return (notme);
> 	}

-1 See above.

> 	vcl.insert -  add a vcl to the switchboard
> 	vcl.remove -  remove a vcl from the switchboard
> 	vcl.replace - atomic add+remove

-1

all of this can be done with labels, we should not go down this path.

> 	There can be any number of "library VCLs" loaded with
> 	"vcl.library" containing only backend/directors
> 	and ACLs (for now).

+1

> How are conflicting selectors resolved ?  In the above examples
> I put "mine(100)" as a way to assign priorities.  Better ideas ?

Same story as above: What if prio 100 is used more than once?

> I'm slightly concerned about the rebuilding/reconfiguration
> of the merged decision data structure when there are many VCL's.

Yes, that!

> Nobody argued for using regular expressions, which I suspect was
> partly a healthy respect for implementing the merge, and partly
> because those fields are not just strings (%xx, case-insensitity,
> I18N DNS etc.)

See above

> It seem obvious to allow multiple selectors on each of the two
> fields, and to give them "or" semantic, so that a single vcl_scope{}
> can match multiple domains and/or multiple urls.
> 
> But assuming the two fields (host+url) inside the selector have
> "and" semantics, I think we should also allow multiple vcl_scope{}
> per VCL, so that a single VCL can handle:

You are starting to go down exactly the rabbit hole I was referring to and it is 
not a nice place.

> I wonder if host+url is too restricive?

It is.

> I can imagine, but dont know the relevance of, also selecting on
> user-agent and particular cookies being present or absent, but with
> the escape-mechanism, we can collect real-world experience before
> we decide that.

 From my perspective, we do not need to do that to collect real-world 
experience, we at UPLEX have been generating/templating "switchboard code" (we 
usually called it "branch code") for years and I can tell you with confidence 
that whatever generalization we came up with, there have always been exceptions 
which needed custom code outside the templating or a second templated processing 
stage, like:

- run common VCL code before the match for efficiency, resilience and separation
   of concerns ("global" ops authority vs. authority over specific hosts)

- rewrite host and/or url before the match (same reasons)

- regex match as a second stage, either for simplicity/maintainability or
   because host names are actually dynamic

Let us not forget that the key strength of Varnish-Cache is the flexibility of 
VCL. The proposed paradigm goes away from "write the code" towards "config 
file". I say yes to config file (declarative style), but only in the form of a 
templater generating VCL.

> B) "Plain backends are too plain"

My take on this is that we should start with the basic infrastructure which is 
needed by existing implementations in-use today:

* reference backends across vcls (see A) )
* share counters across backends
* cross-director notification ("policy" director vs. "defining" director)


> C) Concrete proposal

+1 overall

> When the client opens a segment, it links a unique filename to that
> file, so the inode link-count increases, and it removes that filename
> again when the client no longer needs any data in that segment.

NB: This kills an operations scenario with shared, read-only mounted file 
systems, as used by multiple containers of a pod in k8s. With this change, the 
"logging container" would need write access to the shared file system, which 
takes down a security isolation we have today.

Please remind me why it was that we need this mechanism? Could we not just make 
varnishd always use unique file names and simply unlink the oldest after having 
rolled over a configurable number of files?

This way, old files will remain available for "varnishlog -d" in a predictable 
manner, and vsl clients still having them open will keep their file handle until 
they are done.

With this model, I would think that VSL clients do not rewrite write access to 
the file system VSM resides on.


Thank you for your work, phk, and sorry again for the long delay. This design 
work is important and my failure to respond earlier is totally unrelated to the 
topic.

Nils

*1) https://varnish-cache.org/tutorials/vmod_sets.html

From scan-admin at coverity.com  Mon Jun 30 12:00:21 2025
From: scan-admin at coverity.com (scan-admin at coverity.com)
Date: Mon, 30 Jun 2025 12:00:21 +0000 (UTC)
Subject: Coverity Scan: Analysis completed for varnish
Message-ID: <68627c5518432_1319fd2c80f4a439a814320@prd-scan-dashboard-0.mail>

An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/varnish-dev/attachments/20250630/9d0fb738/attachment.html>