[master] 9469d95 Update with ESI+GZIP section

Poul-Henning Kamp phk at varnish-cache.org
Mon Jan 24 14:48:33 CET 2011

commit 9469d9524c32a137852196600d62874401d5f19c
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Mon Jan 24 13:48:18 2011 +0000

    Update with ESI+GZIP section

diff --git a/doc/sphinx/phk/gzip.rst b/doc/sphinx/phk/gzip.rst
index 77ba94f..5d00b21 100644
--- a/doc/sphinx/phk/gzip.rst
+++ b/doc/sphinx/phk/gzip.rst
@@ -36,11 +36,13 @@ hash lookup.
 Varnish will not do any other types of compressions than gzip, in particular
 we will not do deflate, as there are browser bugs in that case.
-Before vcl_hit{} is called, the backend requests Accept-Encoding is
+Before vcl_miss{} is called, the backend requests Accept-Encoding is
 always set to:
 	Accept-Encoding: gzip
+Even if this particular client does not support 
 To always entice the backend into sending us gzip'ed content.
 Varnish will not gzip any content on its own (but see below), we trust
@@ -63,6 +65,9 @@ Even if the client does not support gzip, you can force the A-C header
 to "gzip" to save bandwidth between the backend and varnish, varnish will
 gunzip the object before delivering to the client.
+In vcl_miss{} you can remove the "Accept-Encoding: gzip" header, if you
+do not want the backend to gzip this object.
 In vcl_fetch{} two new variables allow you to modify the gzip-ness of
 objects during fetch:
@@ -89,4 +94,70 @@ typical use would be:
+First, note the new syntax for activating ESI:
+	sub vcl_fetch {
+		set beresp.do_esi = true;
+	}
+In theory, and hopefully in practice, all you read above should apply also
+when you enable ESI, if not it is a bug you should report.
+But things are vastly more complicated now.  What happens for
+instance, when the backend sends a gzip'ed object we ESI process
+it and it includes another object which is not gzip'ed, and we want
+to send the result gziped to the client ?
+Things can get really hairy here, so let me explain it in stages.
+Assume we have a ungzipped object we want to ESI process.
+The ESI parser will run through the object looking for the various
+magic strings and produce a byte-stream we call the "VEC" for Varnish
+ESI Codes.
+The VEC contains instructions like "skip 234 bytes", "deliver 12919 bytes",
+"include /foobar", "deliver 122 bytes" etc and it is stored with the
+When we deliver an object, and it has a VEC, special esi-delivery code
+interprets the VEC string and sends the output to the client as ordered.
+When the VEC says "include /foobar" we do what amounts to a restart with
+the new URL and possibly Host: header, and call vcl_recv{} etc.  You
+can tell that you are in an ESI include by examining the 'req.esi_level'
+variable in VCL.
+The ESI-parsed object is stored gzip'ed under the same conditions as
+above:  If the backend sends gzip'ed and VCL did not ask for do_gunzip,
+or if the backend sends ungzip'ed and VCL asked for do_gzip.
+Please note that since we need to insert flush and reset points in
+the gzip file, it will be slightly larger than a normal gzip file of
+the same object.
+When we encounter gzip'ed include objects which should not be, we
+gunzip them, but when we encounter gunzip'ed objects which should
+be, we gzip them, but only at compression level zero.
+So in order to avoid unnecessary work, and in order to get maximum
+compression efficiency, you should:
+	sub vcl_miss {
+		if (object needs ESI processing) {
+			unset bereq.http.accept-encoding;
+		}
+	}
+	sub vcl_fetch {
+		if (object needs ESI processing) {
+			set beresp.do_esi = true;
+			set beresp.do_gzip = true;
+		}
+	}
+So that the backend sends these objects uncompressed to varnish.
+You should also attempt to make sure that all objects which are
+esi:included are gziped, either by making the backend do it or
+by making varnish do it.

More information about the varnish-commit mailing list