Refreshing modified content

[James A. Robinson]

On Thu, Dec 16, 2010 at 12:31, Paulo Paracatu <paulo at aliancaproject.com> wrote:
> If I understood it, the purging method isn't automatic, right? I'd need to
> purge the content everytime it is modified.
> This is kinda stupid... I host more than 10k sites, modifying files
> everytime. If I set a high TTL, the backend will be happy and the webmaster
> will be angry. If I set a low TTL, the webmaster will be happy, but the
> backend will die. Plus, there is no point using a cache if the TTL is low.

In a later post you ask about whether or not varnish could be
configured to send a conditional GET on every request

Basically varnish would be looking up an item in its own cache, seeing
if it had a Last-Modified or ETag header from the backend, and sending
a conditional GET -- if it got an entity back it'd store that entity
as the new version, otherwise serve the old.  I'd be curious if
anyone's put together VCL logic that is capable of that.  It'd be good
to know how to do it.

If it's possible to due, this technique might work well when you are
fronting a backend that is very fast at computing conditional GETs,
e.g., static files that can be examined to see if its inode, size, or
last modified time has changed.  I imagine most people using varnish
have slower backend servers, ones that build dynamic content and
aren't able to respond to conditional GETs any more efficiently than
they could respond to an unqualified GET.

One of the places we use varnish at my dept is fronting a large (half
terabyte of about six million files) static file server.  Instead of
using some form of conditional GET, what we use are cache channels.

  http://tools.ietf.org/id/draft-nottingham-http-cache-channels-01.txt

When various programs are updating the backend server static files
they POST the filepath to our cache channel server.  We have another
program running that monitors the cache channel once a minute for
updates, and when it sees a new entry show up it turns around and
sends a PURGE request to varnish.

Jim

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
James A. Robinson                       jim.robinson at stanford.edu
Stanford University HighWire Press      http://highwire.stanford.edu/
+1 650 7237294 (Work)                   +1 650 7259335 (Fax)