VIP9 - Expanding VCL object support

[Reza Naghibi]

This is another discussion for VIP9. The previous mailing list thread is
linked below [0].

This allows objects to live in req, req.top, and bereq scopes. Currently
objects are global to the VCL and can only be defined in vcl_init.

The driver for this is allowing for multiple VMOD objects to exist in a
single request, each with their own attributes. There is a usecase where we
need to make multiple HTTP requests from VCL to 3rd party services and then
build multiple security related digests on several aspects of the request,
response, and 3rd party services. So having proper objects here would make
the VMOD based solution extremely clean and easy to understand.

I have a branch ready for a PR, but it was requested to have another
discussion. Branch is located here [1] and based on master, 5.0, as of last
week. m00026.vtc shows how the VCL looks [2]. The goal would be to have
this available to VMODs in the next major release, Q1 2017.

I will briefly address several of the concerns that have been brought up:

* Syntax is too verbose. I agree that the first iteration kind of went
overboard with the syntax. Currently, the syntax requires a (scope) style
cast when you define the object and thats it. Please see [2].

* This implementation is completely backwards compatible with how objects
are defined in 4.X and does not require VMOD code to change. Object VMODs
will be immediately available to these new scopes and will still go thru
the _init() _fini() lifecycle.

* Conflicts with VIP1. This VIP is only concerned with exposing PRIV_TASK
and PRIV_TOP into existing objects at the VCL level. I believe VIP1 is
concerned with PRIV_* at the VMOD level. So I believe these are a bit
orthogonal and independent.

* VMOD safety. Im pretty confident that VMOD objects actually allow for
higher levels of reference and memory safety because objects have explicit
_init() and _fini() methods and are passed in a struct for tracking state.
So if you wanted to reference objects from other objects, then you could
implement your own ref counting algorithm ontop of __init(), __finish(),
and the state (or a gc implementation or whatever algorithm you want). I
think this argument is a bit arbitrary because if we look at something as
simple as VCL_STRING, which all VMODs have access to via req/resp headers,
if you were to reference that in a VMOD, it will be freed from under you
and leave you with unsafe code and memory. So any kind of unsafe VMOD
example probably extends to the entire VMOD universe and should not be
limited to this one VIP. Safe coding practices used today in VMODs should
not be abandoned in context of this VIP :)

It might be more helpful to see the compiled VCL code interact with the VRT
to understand how this PR will work. So the C code for m00026.vtc is
located here [3]. Just grep for g0, r0, r1, t0, and b0 to better see how
this is implemented.

[0]
https://www.varnish-cache.org/lists/pipermail/varnish-dev/2016-April/008906.html

[1]
https://github.com/varnishcache/varnish-cache/compare/master...rezan:feature/object_scopes_master

[2]
https://github.com/rezan/varnish-cache/blob/feature/object_scopes_master/bin/varnishtest/tests/m00026.vtc

[3] https://gist.github.com/rezan/76f92f76d31ee2d2105501b63612db43

--
Reza Naghibi
Varnish Software
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/varnish-dev/attachments/20161109/2824aa26/attachment.html>