Streaming and backend conditional requests

[Poul-Henning Kamp]

Sorry for being a bit late to this discussion, I'm still trying to get
through my contractually obligated New Year ritual.

I think the discussion is somewhat muddled by looking too much at
how to move forward, I would far rather discuss where we want to
go in the first place.

My goal, is to have backend-IMS and streaming fetch both be default-on
features.

Usually the way we've done things is one release with $newfeature being
default-off, then make it default-on in the next release, but I am seriously
considering skipping that intermediate step, because it invariably
means VCL changes which become pointless a release later.

So, going directly after the IMS+Streaming situation, there's how I
see the VCL:

Pipe is pipe and nothing changes there.

Once you issue a fetch, (from vcl_pass/vcl_miss) you _always_
end up in vcl_fetch{}, but you may have a synthetic 503 object if we
failed to get a backend connection.

This allows the 503 to be cached (ie: single-backend, no-probe)

By default all backend fetches which can find a candidate for it,
will try to IMS against the backend.  If you do not want that, you
remove the bereq.http.IMS header in vcl_pass{}/vcl_miss{}

In vcl_fetch{} we add a "BOOL fetch_body()" function, so that you
can insist on pulling the entire object from the backend, before
accepting it into the cache, before deciding to deliver it to
the client, or for doing (vmod-) modifications to it before delivery.

If you do not call fetch_body(), only the headers are available in
vcl_fetch{} and the body will be fetched asynchronously and delivered
to any client hitting this object in parallel.

vcl_error{} becomes vcl_synth{}, and you call it like:

	return(synth(202));
	return(synth(578, "You got to be kidding"));
And the arguments go into resp.status and resp.response

vcl_synth{} will only have access to req.* and resp.*, no obj.* or bere*.*

Two issues need resolved:
	Can you cache synth objects ?
	Do they get delivered through vcl_deliver ?

My initial idea was "no & no", but I am willing to be persuaded otherwise,
and Artur may be halfway there already.

The argument for no caching is simply that Varnish is not a webserver
and I'd hate to see people try to turn it into one.

The second is more involved:  Synth replies will not have an obj.*
and therefore delivery is mechanically quite different from an obj.*
based delivery.  Also, we won't be doing gunzip or ESI on synth
replies.

Now, there is at least one inconsistency in the above, and I'm open
for input on how to best solve it:

You cannot change the body of the 503 you get in vcl_fetch{}, unless
you call vcl_synth{}, but then you can't cache that 503...

-- 
Poul-Henning Kamp       | UNIX since Zilog Zeus 3.20
phk at FreeBSD.ORG         | TCP/IP since RFC 956
FreeBSD committer       | BSD since 4.3-tahoe    
Never attribute to malice what can adequately be explained by incompetence.