Poul-Henning Kamp phk at phk.freebsd.dk
Thu Feb 17 21:13:26 CET 2011

In message <4D5D7096.6040701 at uplex.de>, Geoff Simmons writes:

>[...]a new parameter for that. Let's say it's called refresh_ttl

I would probably call it "conditional_timeout", as refresh_ttl could
sound like it would actively refresh things.

>  - In HSH_Lookup(), if an object o is found in the objhead list such
>    that:
>    - o->ttl has elapsed, but o->refresh_ttl has not elapsed, and
>    - o has a Last-Modified and/or ETag header
>    then o is a candidate for refresh (o becomes stale_obj).

yes, (presuming we have weeded out vary-non-matches first)

But what about bans and objects which have had their TTL+grace set
to zero as a means of purging ?

Formally, the backend should DTRT and not 304 these, but that sort of
pressumes the backend know about them.

Imagine the backend with a PHP-bug.  The bad result gets cached,
problem gets fixed, cache gets purged (obj.ttl = 0s), next fetch
uses this object, asks backend IMS ? and the backend, unaware that
the PHP mistake was fixed, checks its database and says "Sure: 304!"

In total there are three cases:

	We should not touch these ever, the current code will do
	ban-processing as part of the search, so that should come

ttl=0, grace=0
	This is a clear cut expiry, but we may still catch it before the
	grim reaper does, we should explicity make sure we do not.

	We should probably have the VRT code zero the conditional_timeout,
	like it does with grace, when ttl is set <= 0.

ttl=0, grace>0
	This is a special case, where the object is expired, but still
	available for grace (you have to first set obj.ttl = 0 and then set
	obj.grace to nonzero, as obj.ttl = 0 also clears obj.grace)
	I think in this case it is a candidate, the admin clearly considers
	the content valid enough to use in an emergency, so it should be
	fair game for IMS also.
	But if we treat conditional_timeout as grace, this follows 
	automatically:  If you set obj.ttl = 0, you have to explicitly
	enable grace and IMS use of the object.
>  - a grace candidate is determined as it is now (independently of
>    refresh_ttl)

Yes, they should be independent.

>  - the cache eviction time for an object (oc->timer_when) becomes
>    o->ttl + max(o->refresh_ttl, HSH_Grace(o->grace))
>That is, an object is evicted when grace or refresh_ttl has expired,
>whichever is later.


We may eventually have to teach the grim reaper about the difference
so that LRU becomes "Least recently used, least useful" or something,
but lets leave that one "for further study"

>The idea has to do with possible clock skew between different backends
>that delivered the same object; and the RFC admits that the 60 seconds
>is arbitrary. But since Varnish tries to account for clock skew, and
>smart admins use NTP, I think we shouldn't worry about it.

Having thought about it: As long as we use the Last-Modified header
provided by the backend in the IMS, it should not be our problem.

But I guess this would apply when the client sends us a constructed
timestamp (ie: not a copy of the backends L-M header).  My bat-sense
is tingling a little bit over this, and I may have to revisit the way
we do IMS from clients to make sure we do not offer a security-wedge.

But you are right: as long as we copy the backends L-M header, we
need not worry.

>but we might consider adding some features in
>the future, for better compliance (and some of it might be useful).

I'm not particular keen on adding stuff nobody is using, just to
be "compliant" with one of the least coherent RFC's we have.  But
useful stuff I'm entirely open for.

>- - For a first implementation, we agreed to put off decisions about
>sharing storage between objects, until we're clear and how best to go
>about it, and just copy the storage from the stale object to the new
>object after a 304 response.


>It was good to see everybody at VUG, hope you're all doing well and
>getting hit often. %^)

Yes, nothing like being able to use your hands to communicate with
also :-)

Poul-Henning Kamp       | UNIX since Zilog Zeus 3.20
phk at FreeBSD.ORG         | TCP/IP since RFC 956
FreeBSD committer       | BSD since 4.3-tahoe    
Never attribute to malice what can adequately be explained by incompetence.

More information about the varnish-dev mailing list