Degraded operation

[Dag-Erling Smørgrav]

What follows is my design proposal for degraded operation, i.e. the
ability to serve expired content when a backend is not responding.

 - The first step is to stop discarding objects when they expire.
   Instead of placing the object on death row, the misnamed
   exp_prefetch() simply marks it as expired.

   If the expired object is never needed again, it will eventually be
   discarded by LRU_DiscardLocked() (or not, if the cache never fills
   up)

 - As a consequence, Varnish will start getting cache hits for expired
   objects.  Therefore, vcl_hit() must be able to inspect the object's
   expired flag and decide whether to serve the expired object or
   fetch a new copy.

 - In order to make that decision, vcl_hit() must be able to figure
   out which backend the object came from in the first place and
   inspect that backend's health metric.

   This opens up a number of interesting possibilities - things like
   "if the backend is starting to get bogged down but not quite dead
   yet, only refresh objects that are more than N seconds past their
   expiry time".

 - Now that vcl_miss() is no longer the only path to a backend
   request, we need to take a closer look at it.  It really performs
   two distinct functions: one of them is to react to a cache miss,
   the other is to prepare a backend request.  We need to separate
   these two functions, so if vcl_hit() decides to fetch a new copy,
   control can pass straight to the prepare-a-backend-request bit
   without passing through the react-to-a-cache-miss bit.

   I would like to name that last part vcl_fetch(), but that name is
   already taken by the verify-the-result-of-the-backend-request bit.
   We should rename vcl_fetch() to something else, though I'm not sure
   what.

 - It would be nice to be able to define global variables that can be
   manipulated in the management interface as well as in VCL; for
   instance, an admin might want a "no-modify" flag that when set,
   causes VCL to never refresh objects which are already in cache.  On
   the other hand, this can be done by loading a different VCL config.

There are still a few gaps:

 - If we try to refetch an expired object, not knowing (yet) that the
   backend is dead, what happens to the old copy of the object?
   Preferably, it should remain in the cache so we can serve it
   instead of the non-existent fresh copy.

 - This does not address prefetching at all - I will have to think
   about how to implement it once the above is in place.

DES
-- 
Dag-Erling Smørgrav
Senior Software Developer
Linpro AS - www.linpro.no