Possible to detect a previous xkey softpurge?

[Dridi Boukelmoune]

On Thu, Sep 3, 2020 at 10:57 AM Batanun B <batanun at hotmail.com> wrote:
>
> Hi,
>
> We sometimes have a problem with the backend using its internal cache for a few seconds too long after something has been updated. We trigger a softpurge (xkey vmod) in varnish, but if someone requests the page again very soon after that, the data that Varnish gets from the backend might be old. In this case, we would like to be able to tell the backend, maybe using an extra header, that it should skip its internal caches and give us the updated content.

You can't really do that, but the closest I can think of would look like this:

    sub vcl_backend_fetch {
        if (bereq.is_bgfetch) {
            # tell the backend to somehow not serve from cache
        }
    }

Since a soft purge artificially moves you inside the grace period, any
hit on those objects would trigger a background fetch.

Arguably, if you use Varnish to cache responses, you might as well
always tell your backend not to serve from cache. Because if a soft
purge moves you inside the grace period, there's no guarantee that the
next hit will happen before the object leaves the grace period. At
this point this will no longer trigger a background fetch...

> But, I'm not sure how to archive this in Varnish. Is it possible to detect that the page requested has been softpurged earlier? If yes, is it also possible to see when that softpurge took place? Because we would only ever need to do this if the softpurge happened less than let's say 30 seconds ago.
>
> And the reason that the backend data might be old after an update is that what we call "the backend" (from a Varnish perspective) is actually a complex setup of services. And there, the update happens in one place, and our "backend" is actually a frontend server that sometimes don't get the information about the update quick enough. I think that the implementation of this system is a bit faulty, but it is what it is, and I would like to use the power of Varnish to handle this, if possible.

Objects are immutable (modulus TTL) once they enter the cache so
there's nothing actionable to record that they were soft-purged.

Dridi