[master] 9469d95 Update with ESI+GZIP section

[Poul-Henning Kamp]

commit 9469d9524c32a137852196600d62874401d5f19c
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Mon Jan 24 13:48:18 2011 +0000

    Update with ESI+GZIP section

diff --git a/doc/sphinx/phk/gzip.rst b/doc/sphinx/phk/gzip.rst
index 77ba94f..5d00b21 100644
--- a/doc/sphinx/phk/gzip.rst
+++ b/doc/sphinx/phk/gzip.rst
@@ -36,11 +36,13 @@ hash lookup.
 Varnish will not do any other types of compressions than gzip, in particular
 we will not do deflate, as there are browser bugs in that case.
 
-Before vcl_hit{} is called, the backend requests Accept-Encoding is
+Before vcl_miss{} is called, the backend requests Accept-Encoding is
 always set to:
 
 	Accept-Encoding: gzip
 
+Even if this particular client does not support 
+
 To always entice the backend into sending us gzip'ed content.
 
 Varnish will not gzip any content on its own (but see below), we trust
@@ -63,6 +65,9 @@ Even if the client does not support gzip, you can force the A-C header
 to "gzip" to save bandwidth between the backend and varnish, varnish will
 gunzip the object before delivering to the client.
 
+In vcl_miss{} you can remove the "Accept-Encoding: gzip" header, if you
+do not want the backend to gzip this object.
+
 In vcl_fetch{} two new variables allow you to modify the gzip-ness of
 objects during fetch:
 
@@ -89,4 +94,70 @@ typical use would be:
 GZIP and ESI
 ------------
 
-TBD.
+First, note the new syntax for activating ESI:
+
+	sub vcl_fetch {
+		set beresp.do_esi = true;
+	}
+
+In theory, and hopefully in practice, all you read above should apply also
+when you enable ESI, if not it is a bug you should report.
+
+But things are vastly more complicated now.  What happens for
+instance, when the backend sends a gzip'ed object we ESI process
+it and it includes another object which is not gzip'ed, and we want
+to send the result gziped to the client ?
+
+Things can get really hairy here, so let me explain it in stages.
+
+Assume we have a ungzipped object we want to ESI process.
+
+The ESI parser will run through the object looking for the various
+magic strings and produce a byte-stream we call the "VEC" for Varnish
+ESI Codes.
+
+The VEC contains instructions like "skip 234 bytes", "deliver 12919 bytes",
+"include /foobar", "deliver 122 bytes" etc and it is stored with the
+object.
+
+When we deliver an object, and it has a VEC, special esi-delivery code
+interprets the VEC string and sends the output to the client as ordered.
+
+When the VEC says "include /foobar" we do what amounts to a restart with
+the new URL and possibly Host: header, and call vcl_recv{} etc.  You
+can tell that you are in an ESI include by examining the 'req.esi_level'
+variable in VCL.
+
+The ESI-parsed object is stored gzip'ed under the same conditions as
+above:  If the backend sends gzip'ed and VCL did not ask for do_gunzip,
+or if the backend sends ungzip'ed and VCL asked for do_gzip.
+
+Please note that since we need to insert flush and reset points in
+the gzip file, it will be slightly larger than a normal gzip file of
+the same object.
+
+When we encounter gzip'ed include objects which should not be, we
+gunzip them, but when we encounter gunzip'ed objects which should
+be, we gzip them, but only at compression level zero.
+
+So in order to avoid unnecessary work, and in order to get maximum
+compression efficiency, you should:
+
+	sub vcl_miss {
+		if (object needs ESI processing) {
+			unset bereq.http.accept-encoding;
+		}
+	}
+
+	sub vcl_fetch {
+		if (object needs ESI processing) {
+			set beresp.do_esi = true;
+			set beresp.do_gzip = true;
+		}
+	}
+
+So that the backend sends these objects uncompressed to varnish.
+
+You should also attempt to make sure that all objects which are
+esi:included are gziped, either by making the backend do it or
+by making varnish do it.