[RFC] Changes to pass mode

[Poul-Henning Kamp]

The original intent was that pass-mode should be able to handle PUT
and POST requests (if invoked from vcl_recv{} and should allow reuse
of the backend connection afterwards.

Judging from the feedback and requests I have received, this is
unnecessary generality.  POST and PUT seems to be heavily tangled
in authentication and cookies which disable caching anyway, so using
pipe-mode for those, does not seem to hurt anybody to any extent
that matters.

Discarding the PUT/POST handling requirement from pass, allows me
to treat pass more or less as a miss, and thus avoid a special
case of the functions for talking to the backend, and it gives us
one of the much asked for facilities: slow-client mitigation for
CGI content done in pass mode, almost for free.

This results in some some subtle and some not so subtle changes,
which I am looking for feedback on.

It will help if you have this dot(1) plot open in a window while
reading the rest of this email:
	http://phk.freebsd.dk/misc/varnish.gif

Take a moment to follow the green/blue paths which are the normal
hit/miss scenarios.

Pass can take two variants in this new world order, from vcl_recv{}
or from anywhere else.

If vcl_recv{} does pass, an object is created which is not inserted
into the hash table, and it is marked as not cacheable.  (The object
will not be inserted in the hash, even if vcl_fetch() invokes "insert".)

We then proceed as a normal cache-miss.

Since the object is not inserted in the hash, the next client to
ask for this url will not find this object and unless that client
also gets a pass in vcl_recv{}, normal processing will happen.

The upshot of this is, that certain clients can be forced to do
a pass in vcl_recv{} without forcing all clients to do this.

If pass is invoked from vcl_miss{} or vcl_fetch{}, an entry
will be inserted into the hash marking this object as a "obj.pass"
object which cannot be cached.

If vcl_hit{} invokes pass, the request will be serviced as a pass,
even if we had found a good cached object.

If vcl_hit{} finds a "obj.pass" hit in the hash, the request
will also be handled as a pass.

The other change
----------------

The other change to be aware of, is that the entire object is
fetched from the backend before vcl_fetch() is called, previously
we only had the headers at that point.

This allows us to detect backend errors in vcl_fetch{}, and once we
implement "retry lookup" we can aim the request at another backend
or whatever error-recovery we prefer.

It also paves the way for gzip'ing things in vcl_fetch() later on.

Does this make sense ?

Any better ideas ?

Does anybody even read this mail-list anymore ?  :-)

Poul-Henning

PS: if you want to see the current state diagram, the instructions
for how to create it is at the top of bin/varnishd/cache_center.c


-- 
Poul-Henning Kamp       | UNIX since Zilog Zeus 3.20
phk at FreeBSD.ORG         | TCP/IP since RFC 956
FreeBSD committer       | BSD since 4.3-tahoe
Never attribute to malice what can adequately be explained by incompetence.