Request/response body access functions.

[Arianna Aondio]

Hi everyone,
this email is about body transformations.
Both Apache and Nginx allow transformations, such as regular
expression with substitution, on request and response bodies, in my
personal opinion implementing such a thing in varnish would add value
to our software.

What do we have so far?
request body transformations have already been discussed at Hamburg
VDD and later on, but patches never had the chance to become part of
the varnish core.

Notes form Hamburg VDD:
------------------------------------------------------------------------------------------------------------------------------
* Request body functions, review of suggested design. (arianna)
  phk brought up the issue of response body access. this ties it to
body access in VCL.

// req.body in vcl is immutable
// you need a VMOD VFP to modify the body going to the backend (see TODO above)
in vcl_recv{}
  req.cache_body(100k); // fails request if longer req.body
  req.bodylen = N;      // actual length
  hash_data(req.body);  // takes BLOB, requires STRING -> BLOB silent conversion
                        // can be microoptimized by VCC to walk storage chunks
  regsub(std.tostring(req.body), "..", "..");

IDEA: make BLOBS iterators?  // phk needs to ponder
-----------------------------------------------------------------------------------------------------------------------------

Since no more input has been given regarding this topic, I have
implemented a vmod that allows the user to do different operations
using the request body.
The vmod build against Varnish 4, but it can be improved when Varnish
4.1 will be released; at the moment the code is a little bit redundant
due to the presence, in branch 4.0, of VRB_Cache() not able to handle
properly POST body larger than the provided size limitation( this has
been fixed in master branch).
Please feel free to use and test the
vmod(https://github.com/aondio/libvmod-bodyaccess/).

At this point it comes natural to start thinking about response bodies:
the idea is to modify or do any kind of operations on the response
body before the object is in the cache, this means if we want to be
able to make transformations on the response body those have to be
done at fetch time.
When VBF_Fetch is triggered a new object is fetched from backend, at
that point a "chain" is initialized and every link of this chain
contributes to give shape to the final response, for example:

VBF_Fetch ---> GZIP---> suck bytes

In this simple situation when a new object is fetched the GZIP "link"
will actually gziped the "sucked" bytes and then deliver them.
The same logic applies for UNGZIP and ESI( |GZIP|UNGZIP).
I'd like to have a vmod that can insert itself in the fetch chain,
modify the response body as required and return without making varnish
crash or start behaving strangely.
In order to make it possible some changes to the varnish core need to
be done, but I don't have any patches for those changes because I'd
first like to hear what you think about this idea.

Comments are much appreciated.
-- 
Arianna Aondio
Software Developer | Varnish Software AS
Mobile: +47 980 62 619

We Make Websites Fly
www.varnish-software.com