[PATCH] Streaming and Content-Length

[Poul-Henning Kamp]

In message <53AC78E2.8090505 at schokola.de>, Nils Goroll writes:

>here's the first set of patches to address #1506. These do not cover
>Range-Requests yet, but these should be easy to address once we got this first
>set in.

I've looked at this, but it touches on a number of related areas which
I feel calls for a more general solution.

The present issue is that we don't want to throw a perfectly good
Content-Length from the backend away, if we can send it to the
client while we are streaming an object still being fetched.

But if we gzip, gunzip or ESI process the object, that C-L must not
be passed to the client, since it will (may really) not match the
data they get.

The general issue is:  Who controls the Content-Length header and what
relation, if any, does have to obj->len.

The answer to the first part is that we have to ask the fetch and
delivery filters that question: nobody else knows, and in the future
there may be such filters in vmods which we don't even know about,
until we ask them.

So the solution is to pile up the fetch-pipeline before calling
vcl_backend_response{} and once v_b_r{] returns, we ask the pipeline
if the C-L (if the backend sent one) is vetoed by the pipeline.

For instance the gzip and gunzip VFP's would both veto it.

Similarly, we have to ask the delivery pipeline, so that VDP::gunzip
can veto the C-L because it will not match the result.

That is going to be in 4.1

So far, so good, what about the second half of the question: obj->len ?

If we get a uncompressed object from the backend and VFP::gzip it,
we do actually know what C-L to send in case we VDP::gunzip it later,
if only we had a place to store it in the meantime.

This sounds like a pathological case, but it is actually indicative
for a large class of things VMODs might do to objects, where a VFP
and VDP will work in tandem, if only they could talk to each other.

One very prominent example is ESI, where we produce a bytestring
which tells us how to rip the object apart for ESI processing.
Today we store that ESI string totally separate directly in struct
object, because there is only that single "out-of-band" information
at this time.

But down the road, not sure when, I'll add a generalized facility
so that VFP's can attach attributes to objects, and they will be
appended to the obj.body encoded somehow.

That will allow VFP:gzip and VFP:testgunzip to store the "other"
length as an attribute which VDP:gunzip can retrive and emit
as a C-L.

And it will allow VFP::ESI to store two attributes, one with
all the includes, and one with the instruction stream, so that
we can fire of all the includes in parallel while we compose
the ESI object.

The caveat is that the attributes will only be available once the
full object body has been fetched, so the VDP's either need to have
a fall back strategy.  For instance VDP::gunzip will just veto any
C-L and fall back to chunked encoding if there is no attribute
available with the C-L to use.

This may or may not be in 4.1

-- 
Poul-Henning Kamp       | UNIX since Zilog Zeus 3.20
phk at FreeBSD.ORG         | TCP/IP since RFC 956
FreeBSD committer       | BSD since 4.3-tahoe    
Never attribute to malice what can adequately be explained by incompetence.