Using Varnish as a backend for another Varnish server, with ESIs

[Kolychev, Aleksey]

Are you load-balancing between the two varnish instances? If you are, you should check if you're hashing the cache keys properly, e.g., not sending the same response to the other varnish instance that you'd send to all other clients, so that your outer varnish instance doesn't serve cached pages with unparsed ESI tags to clients.

You could try doing something along the lines of -

sub vcl_recv {
  if(client.ip ~ other-varnish) {
    set req.http.X-Via-Varnish = 1;
  }
}

sub vcl_hash {
  set req.hash += req.http.X-Via-Varnish;
}

sub vcl_fetch {
  if (beresp.http.X-Meta-Has-Esi) {
      unset beresp.http.X-Meta-Has-Esi;
  } else {
      esi;
  }
}

(Disclaimer - I didn't test this, and cannot be held responsible for syntax errors)

--
Lex Kolychev
Application and Systems Architect
Internet Technology Dept.
Martha Stewart Living Omnimedia
Ph:   212-827-8948


From: Pax Dickinson <pax at businessinsider.com<mailto:pax at businessinsider.com>>
Date: August 17, 2011 2:38:16 PM EDT
To: "varnish-misc at varnish-cache.org<mailto:varnish-misc at varnish-cache.org>" <varnish-misc at varnish-cache.org<mailto:varnish-misc at varnish-cache.org>>
Subject: Using Varnish as a backend for another Varnish server, with ESIs

Hi,

I'm attempting to use two varnish servers as the preferred caching
backend for each other, falling back to my apache backends if neither
varnish server has the content in cache. I'm doing it to reduce my
backend traffic and keep my caches hot more efficiently than having
each varnish server totally separate, which is what I do now. Also
this will scale better if the day comes that I need a third server.

I match against an ACL and if a request comes from anywhere other than
my other varnish server, I set the varnish backend. This is working
fine, except my edge side includes need to be processed only by the
server that received the original request. If the secondary server
were to process and include my ESIs at that point they would be
returned to the primary server and become part of its cached copy, and
that defeats the purpose of having ESIs.

It seems that it should be as simple as:

sub vcl_recv {
  if (!client.ip ~ othervarnish) {
      set req.backend = varnish;
  }
}

sub vcl_fetch {
  if (beresp.http.X-Meta-Has-Esi && !client.ip ~ othervarnish) {
      esi;
  }
}

BUT.. If I call it as above, ESIs work and are processed correctly by
whichever server received the request first. However, when hitting the
secondary server and getting a cached result the ESIs are not
processed, vcl_fetch is never called, and the esi tags remain in the
result. Also, it seems that the esi function can only be called in
vcl_fetch. In vcl_deliver it does nothing.

So for example:

The initial request works:

Request ---> VarnishA MISS --> VarnishB MISS --> Backend (success with
esi parsed)

However subsequent requests look like this:

New Request ---> VarnishA HIT (success with esi parsed)
New Request ---> VarnishB HIT (fail, esi will not be parsed)

Is there a way to make this work? Or is using varnish as a backend for
another varnish incompatible with edge side includes? I'm using
varnish 2.1.5, is this possible in 3.0?

Thanks..

--
Pax Dickinson
Chief Architect
Business Insider
257 Park Avenue South, 13th floor
New York, NY 10010
pax at businessinsider.com<mailto:pax at businessinsider.com>

_______________________________________________
varnish-misc mailing list
varnish-misc at varnish-cache.org
https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc


________________________________
Don't miss a brand new season of the Emmy award-winning "Martha Stewart Show", followed by the all-new series "Emeril's Table"...Beginning September 26, only on Hallmark Channel.

Find Martha Stewart on Facebook: http://facebook.com/MarthaStewartLiving

Follow Martha Stewart on Twitter: http://twitter.com/MarthaStewart

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/varnish-misc/attachments/20110817/24840579/attachment-0003.html>