Testing Varnish

[Poul-Henning Kamp]

In message <E077482A-817A-4F54-A91A-C74AA34FD7FC at yahoo-inc.com>, Mark Nottingha
m writes:

>> Varnish is not a proxy in the RFC2616 sense.  Varnish
>> is an origin server.  An origin server which gets its contents
>> using HTTP, but an origin server nontheless.
>
>That's a gateway, a.k.a. surrogate, a.k.a. reverse proxy. 

In RFC2616 context, Varnish is an origin server and a client, but
it is not a proxy or a gateway.  Varnish is designed and built to
be able to violate the semantic transparancy requirement for both
of those.

The name Varnish derives from the dictionary definition that says
"To give a deceptively good appearance" (...to a CMS system).

The entire point of Varnish is to make up for shortcomings in CMS
systems and these shortcomings are not limited merely to speed.

Varnish may be tasked with fixing up object lifetimes or anything
else the site administrator is unhappy with.

While varnish speaks "state of the art HTTP" on the client side,
it may have to use "pidgin HTTP" with the backend.  RFC2616 does
not even begin to address a situation like that.

For instance section 14.15 says:

   The Content-MD5 header field [...] Only origin servers or clients MAY
   generate the Content-MD5 header field; proxies and gateways MUST NOT 
   generate it, [...]

This is a perfect example of what Varnish might be asked to do: if
the CMS system does not or can not produce Content-MD5 and the site
ovner wants these, Varnish should be able to add them.

Another example is that one of the hottest wishlist items for Varnish
version 2 is ESI(-like) facilities for content composition, there
is no way RFC2616 can allow us to do that, while sailing under a
"gateway" flag.

I want to stress, that this non-adherence to standard is not based
on disrespect for RFC2616 on our part.

The RFC2616 authors tacitly assume that a web-server can be fixed
or improved at will.  Unfortunately experience has shown that other
parts of CMS systems are far more critical than the webserver part
and therefore many sites which would love to, do not have that
option.

Therefore we are trying fill a need the authors of RFC2616 did not
foresee:  The need to put a glossy finish over a rubbish origin
server to make it look good to the clients.

>My concerns are mostly around interoperability with other devices, and  
>controlling how Varnish caches things.

All of "policy" or "behaviour" aspects of Varnish are designed to
happen in the VCL programming language.

The default VCL code is intended to Do The Right Thing, so that a
site administrator hit my abnormal traffic at 2:20AM can kick in a
Varnish box, point it at the backend and have it cope with the spike
and revisit the situation when he wakes up next time.

Therefore Varnish will look a lot like a proxy or gateway when you
look at it first time.  But hopefully it will be much more than
that to you, once you get to know it better.

All that however, does not change the fact that in the separate
roles of client and origin server, Varnish should comply with
RFC2616 to the extent possible, reasonable and productive.

Poul-Henning

PS: At this point, VCL is nowhere as expressive as we dream about,
but that is merely a matter of time, ideas and code.

-- 
Poul-Henning Kamp       | UNIX since Zilog Zeus 3.20
phk at FreeBSD.ORG         | TCP/IP since RFC 956
FreeBSD committer       | BSD since 4.3-tahoe    
Never attribute to malice what can adequately be explained by incompetence.