draft VDD17 minutes

[Poul-Henning Kamp]

I'm resending these, they're the subject of todays bugwash...


Minutes of the 20171009 VDD
===========================

This VDD was a small, by-invitation-only VDD to attempt to nail down
the big changes in the 201803 release cycle.

Participants:	phk, slink, geoff, martin, daghf

Location: @phk, Slagelse, Denmark

Type specific methods
---------------------

An attempt to enable the VCL syntax look more oo-ish/python-ish with
a heuristic definition of "per-type-methods".

Strawman:

	vmod1.vcc:

		$Function REAL foo(STRING, INT)

	vmod2.vcc:

		$Function BACKEND bar(REAL, STRING)

	Today these would combine as:

		bereq.backend = vmod2.bar(vmod1.foo("something", 12), "other");

	With the new syntax it could also be written as:

		bereq.backend = vmod1.foo("something", 12)->vmod2.bar("other");

The '->' takes as left-hand side an expression, and as right hand side
a function or procedure call where the first non-PRIV argument must match
the type of the left-hand expression (subject to automatic foldings).

We introduce the new '->' operator for this because using the '.' is at
least confusing:

	bereq.backend = vmod1.foo("something", 12).vmod2.bar("other");

We talked about changing to ':' or '::' for name scoping instead,
but decided it would break too much VCL:

	bereq.backend = vmod1::foo("something", 12).vmod2::bar("other");

This is VCL4.0 compatible, but should cause a minor bumb (VCL4.1)

VMOD Function pointers
----------------------

Enable VMODs calling of functions determined at VCC-time.

Strawman:

	vmod1.vcc:

		$Type BOOL match_func(STRINGS)

		$Function INT count_req_x(match_func)

	vmod2.vcc:

		$Function BOOL match_cookie(STRINGS)

	vcl:

		sub vcl_recv {
			if (vmod1.count_req_x(vmod2.match_cookie) > 3)) {
				return(error, 400);
			}
		}

At this time VCL functions have no return types, but when they grow that
at a future date, they could also be used.

VMOD importing VMOD
-------------------

Allow vmods to call other vmods

Strawman:

	vmod1.vcc:

		$Object foo(STRING)

		$Method bar(int)

	vmod2.vcc:

		$Import vmod1 [from "..."]

		$Function INT fnord(vmod1.foo, STRING)

	vmod2.c:

		INT
		vmod_fnord(VRT_CTX, VMOD1_FOO foo, STRING str)
		{
			[...]
			VMOD1_FOO_BAR(foo, 12)
			[...]
		}

This one has a number of tricky issues, for instance VCC needs to
create wrapper functions when there are PRIV arguments etc.

Current use-case examples are not particularly strong.

vcl_init object consistency
---------------------------

Today objects can be left uncreated in vcl_init{} with conditional constructs:

	sub vcl_init {
		if (something) {
			new foo = vmod.bar();
		}
	}

This should be legal, and all code should guard against NULL object pointers.

However, leaving an object uninitialized this way, particularly in deep
if-elif-elif... constructs is also a typical error, so we should by
default complain about any objects not created at the end of vcl_init{}
(probably controlled by the vcc_err_unref parameter)

The check that all objects have been initialized would happen at the tail
end of VGC_Load() (== VCL_EVENT_LOAD), and thus it will be vcl.load which
spots the problem and fails.

XXX: Can VCL cold->warm transition also fail due to changed conditional excution?

XXX: Still need to fix _fini destruction order (#2297)


In re #1799
-----------

Consider a VFP/VDP style list of "veto functions" called in sequence
under lock from HSH_Lookup()

	BOOL veto(struct objcore *))

Default list "rfc2616 vary" (better ? "ttl vary")

This will allow VMODS implementing things like cache-key:

	http://httpwg.org/http-extensions/key.html

VMOD object references
----------------------

Vmods must not hold references, explicit or implicit, to foreign objects
outside their activation.

This means that arguments to VMOD functions in particular strings and
backends only can be considered valid during the call of the function.

201803 release VCL
------------------

The goal is to support both VCL4.0 and new VCL version, selecting on the
"vcl" declaration in the VCL.

STRING_LIST implementation
--------------------------

Change from varargs to

	struct string_list {
		int argc;
		const char * const *argv;
	}

VCC knows how many elements will be required, and the struct will be
allocated on the stack of the current context.

Workspace max used VSL
----------------------

Record final WS usage and high-water mark in VSL, use varnishhist()
to ponder.

New WS_highwater() function needed to record size of reserve-free
and maybe lost-to-snapshots high water.

Extensible workspaces
---------------------

New parameter, default off, enables extensible workspaces.  When out
of workspace, new slabs of WS are malloc'ed and links to the old
replaced WS.  On fini walk list and free(3).  Size of extension
segments need to be thought about/decided.

VMODs should check for failure
------------------------------

Vmod functions should check early if VCL has already failed, and
return cheap compatible return value without doing work.

vrt.h should provide macro for this, something like:

	#define ALREADY_GOING_DOWN_IN_FLAMES(x) \
	do { \
		if(*ctx->handling) \
			return(x); \
	} while(0)

VCL variables
-------------

Strawman:

	var TYPE name;

	var TYPE <scope>.var.name; # req.var.foo, bereq.var.bar, ...

Does _not_ allow for VMOD objects which are created with "new".

Smaller scope VCL objects
-------------------------

Allow "new ..." outside vcl_init{}

This affects the arguments to the object creator functions, in particular
there will be no CLI available to whine to.

Drop PRIV_CALL
--------------

PRIV_CALL was more or less entirely invented so that a regexp VMOD
had some place to store the compiled regexp, at a time when we did
not have VMOD objects.  Only in-tree user is std.fileread() which
caches the file opened.

The tacit assumption is that the argument is constant so that the
cached derivation of it can be reused, this prevents usage such as:

	std.fileread(req.url)

With proper vmod object usage, and more dynamic object lifetimes
this could be possible:

	vcl_init {
		new motd = std.fileread("/etc/motd");
	}

	vcl_recv {
		set req.http.motd = motd.read();
		set req.http.bad = std.fileread("/etc/passwd").read();
	}

The last line depends on allowing VMOD object creation outside of vcl_init{}.

VMOD polymorphism
-----------------

Make VCC responsible for picking best prototype:

	vmod1.vcc:

		$Function INT foo(STRING)
		$Function INT foo(BACKEND)
		$Function INT foo(STRING_LIST)

XXX: Not sure if we should also allow return value to vary.
-- 
Poul-Henning Kamp       | UNIX since Zilog Zeus 3.20
phk at FreeBSD.ORG         | TCP/IP since RFC 956
FreeBSD committer       | BSD since 4.3-tahoe
Never attribute to malice what can adequately be explained by incompetence.