Expanding VCL object support

[Reza Naghibi]

Below are some thoughts and a prototype patch for expanding object support.
Today, objects are limited to global objects which have a lifetime of the
entire VCL. I feel its useful to have objects which can be created during
the request and their scope is limited to that request. When the request is
done, the objects are finalized.

The driver for this is creating a new curl/http vmod (I have several other
vmods in mind which would benefit from this). When making curl requests
from VCL, you may want to have multiple outstanding (async) requests, so we
need to have the ability to encapsulate these requests in isolated,
independent, and request scoped objects. Another use case is creating
simple type objects, like VCL_STRING, VCL_INT, and VCL_BLOB. We can wrap
these types into an object and we now have VCL variables which don't
require conversion into headers and we can define them on the fly per
request. This will likely open the door for some very interesting VCL
functionality :)

Here is a VCL snippet which compiles and works with the patch and uses my
new libvmod_types [0].

---
import types;

sub vcl_init
{
  //global objects, these are unchanged from 4.0
  new s = types.string("Hello!");
  new reqs = types.integer(0);
}

sub vcl_recv
{
  //new req scoped objects
  new req.var.slocal = types.string("Request scoped string");
  new req.var.s2 = types.string("request string two");
  new req.var.count = types.integer(1);
}

sub vcl_backend_fetch
{
  //new bereq scoped objects
  new bereq.var.sbe = types.string("berequest string v1");
  set bereq.http.sbe = bereq.var.sbe.value();

  bereq.var.sbe.set("berequest string v2");
  set bereq.http.sbe2 = bereq.var.sbe.value();
}

sub vcl_deliver
{
  //referencing a mix of global and req scoped objects
  set resp.http.X-s = s.value();
  set resp.http.X-s-length = s.length();

  set resp.http.X-slocal = req.var.slocal.value();
  set resp.http.X-slocal-length = req.var.slocal.length();

  req.var.count.increment(10);
  set resp.http.count = req.var.count.value();
  set resp.http.reqs = reqs.increment_get(1);
}
---

The theoretical curl/http example:

---
import http;

sub vcl_recv
{
  //http request #1
  new req.var.h1 = http.request();
  req.var.h1.set_header("foo", "bar");
  req.var.h1.set_url("POST", "http://host1/blah?ok=true");
  req.var.h1.send();

  //http request #2 (we dont read it so its async)
  new req.var.h2 = http.request();
  req.var.h2.set_url("GET", "http://host2/ping");
  req.var.h2.send();
}

sub vcl_deliver
{
  //reference and read http request #1 and block for result
  set resp.http.X-test-response-code = req.var.h1.get_response_code();
}
---

I left the legacy global objects alone in code and syntax. I introduced 2
new variable name scopes: req.var.* and bereq.var.*. This is completely
cosmetic as these variables can still be request scoped without the
(be)req.var prefix. However, the reason for adding it is to give the user
some kind of indication that their variable is tied to a frontend, backend,
or global scope. Otherwise I have the feeling having a bunch of un-prefixed
variables throwing vcc scope errors when used incorrectly will be confusing.

Also, the implementation is fairly simple because I piggybacked on the
vmod/vrt priv_task implementation. Request scoped objects are basically
given a shimmed struct vmod_priv. I had to jump thru a few small hoops in
vcc code to get the priv->priv to cast into an actual struct that the VMOD
expects. This may or may not be related to VIP#1, but it would be cleaner
to move objects to something more priv like than trying to pass in an
explicit struct. However, for the patch, I kept the object interface the
same and made use of the previously mentioned vcc/vrt shims.

The patch is enough to have the examples work and give you guys an idea of
how it would work. I wanted to get some feedback before spending more time
on this. Its based off of this commit [1], so feel free to comment on
github if you want.


[0] https://github.com/rezan/libvmod-types
[1]
https://github.com/rezan/varnish-cache/commit/b547bd9ad2fca9db1ef17ee73b8e9b7df9950c34

Thanks!

--
Reza Naghibi
Varnish Software
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/varnish-dev/attachments/20160406/9af76bcc/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: 0001-Expanded-object-support.patch
Type: text/x-patch
Size: 6176 bytes
Desc: not available
URL: <https://www.varnish-cache.org/lists/pipermail/varnish-dev/attachments/20160406/9af76bcc/attachment.bin>