Using Varnish as a backend for another Varnish server, with ESIs

[Pax Dickinson]

Hi,

I'm attempting to use two varnish servers as the preferred caching
backend for each other, falling back to my apache backends if neither
varnish server has the content in cache. I'm doing it to reduce my
backend traffic and keep my caches hot more efficiently than having
each varnish server totally separate, which is what I do now. Also
this will scale better if the day comes that I need a third server.

I match against an ACL and if a request comes from anywhere other than
my other varnish server, I set the varnish backend. This is working
fine, except my edge side includes need to be processed only by the
server that received the original request. If the secondary server
were to process and include my ESIs at that point they would be
returned to the primary server and become part of its cached copy, and
that defeats the purpose of having ESIs.

It seems that it should be as simple as:

sub vcl_recv {
   if (!client.ip ~ othervarnish) {
       set req.backend = varnish;
   }
}

sub vcl_fetch {
   if (beresp.http.X-Meta-Has-Esi && !client.ip ~ othervarnish) {
       esi;
   }
}

BUT.. If I call it as above, ESIs work and are processed correctly by
whichever server received the request first. However, when hitting the
secondary server and getting a cached result the ESIs are not
processed, vcl_fetch is never called, and the esi tags remain in the
result. Also, it seems that the esi function can only be called in
vcl_fetch. In vcl_deliver it does nothing.

So for example:

The initial request works:

Request ---> VarnishA MISS --> VarnishB MISS --> Backend (success with
esi parsed)

However subsequent requests look like this:

New Request ---> VarnishA HIT (success with esi parsed)
New Request ---> VarnishB HIT (fail, esi will not be parsed)

Is there a way to make this work? Or is using varnish as a backend for
another varnish incompatible with edge side includes? I'm using
varnish 2.1.5, is this possible in 3.0?

Thanks..

-- 
Pax Dickinson
Chief Architect
Business Insider
257 Park Avenue South, 13th floor
New York, NY 10010
pax at businessinsider.com