If-Modified-Since

[Geoff Simmons]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

On 02/17/11 09:13 PM, Poul-Henning Kamp wrote:
> In message <4D5D7096.6040701 at uplex.de>, Geoff Simmons writes:
> 
>> [...]a new parameter for that. Let's say it's called refresh_ttl
> 
> I would probably call it "conditional_timeout", as refresh_ttl could
> sound like it would actively refresh things.

All right, conditional_timeout it will be. And I assume that the
per-object attribute should have the same name.

While we're on the subject of naming -- we're adding two stats counters
that currently have names that are probably confusing:

cache_refresh_hits -- conditional backend requests that resulted in 304
cache_refresh_misses -- such requests that resulted in 200

Maybe they should be something like:

cache_conditional_not_modified
cache_conditional_refresh

... respectively. We should get some clarity on what Varnish should do
if it gets another response besides 200 or 304 from the backend; but
let's save that for another mail, this one's getting long enough. %^)

Also, we're currently updating the per-object hits counter when a 304 is
returned for a conditional request; but perhaps we also need the two new
stats counters per object, rather than counting this event as a hit.

And to come back to something that I brought up in IRC last week -- when
we merge data from stale_obj into the new object after a 304 response,
that has the effect of resetting the hits counter back to 0 (which of
course makes incrementing the hits counter superfluous). The client at
that point sees the new object with new XID for the URL, which after the
304 backend response has no hits.

Intuitions don't seem to be very clear about whether this is the right
idea. On the one hand, it really is a new object with no hits; but on
the other hand, the cache did find an object that is verified by the
backend as unchanged, and the fact that Varnish is using a new object in
its place is a sort of formality, that might be unimportant to whoever's
looking at obj.hits.

I guess that what matters is what people are using obj.hits for in
real-life VCL. Looking at it that way, it's probably better to copy over
the hits counter, even though Varnish thinks of it internally as a new
object, so that obj.hits keeps doing what people expect.

>>  - In HSH_Lookup(), if an object o is found in the objhead list such
>>    that:
>>    - o->ttl has elapsed, but o->refresh_ttl has not elapsed, and
>>    - o has a Last-Modified and/or ETag header
>>    then o is a candidate for refresh (o becomes stale_obj).
> 
> yes, (presuming we have weeded out vary-non-matches first)
> 
> But what about bans and objects which have had their TTL+grace set
> to zero as a means of purging ?
> 
> Formally, the backend should DTRT and not 304 these, but that sort of
> pressumes the backend know about them.
> 
> Imagine the backend with a PHP-bug.  The bad result gets cached,
> problem gets fixed, cache gets purged (obj.ttl = 0s), next fetch
> uses this object, asks backend IMS ? and the backend, unaware that
> the PHP mistake was fixed, checks its database and says "Sure: 304!"
> 
> In total there are three cases:
> 
> bans
> 	We should not touch these ever, the current code will do
> 	ban-processing as part of the search, so that should come
> 	automatically.

In HSH_Lookup(), in the loop through oh->objcs, we're checking for the
candidate object after the check against BAN_CheckObject(), so as not to
pick up an object with a ban. (It also comes after the check against Vary.)

> ttl=0, grace=0
> 	This is a clear cut expiry, but we may still catch it before the
> 	grim reaper does, we should explicity make sure we do not.

OK, we don't have anything to check for this case, so in our current
code it might in fact pick up such an object. All right, then we'll
explicitly rule this out.

> 	We should probably have the VRT code zero the conditional_timeout,
> 	like it does with grace, when ttl is set <= 0.

Ah, OK.

> ttl=0, grace>0
> 	This is a special case, where the object is expired, but still
> 	available for grace (you have to first set obj.ttl = 0 and then set
> 	obj.grace to nonzero, as obj.ttl = 0 also clears obj.grace)
> 	I think in this case it is a candidate, the admin clearly considers
> 	the content valid enough to use in an emergency, so it should be
> 	fair game for IMS also.
> 	But if we treat conditional_timeout as grace, this follows 
> 	automatically:  If you set obj.ttl = 0, you have to explicitly
> 	enable grace and IMS use of the object.
> 	
>>  - a grace candidate is determined as it is now (independently of
>>    refresh_ttl)
> 
> Yes, they should be independent.

I believe (have to make sure with a test) that in this situation, we
attempt the conditional request. That is, if the TTL has expired, but
neither conditional_timeout nor grace have expired, then the object is a
candidate for the IMS request.

Hm, I really need to test that, because I think it means that, in
situations where grace mode had previously used (either there is a busy
object or the backend is unhealthy), we might now attempt the
conditional request instead of sending back the grace candidate. If so,
then it's changing Varnish's behavior.

Maybe that's the better solution, and since you're allowing changes in
3.0 that break backward compatibility, it might be OK. But we need to
make sure of what happens, so we'll test to find out.


Best,
Geoff
- -- 
** * * UPLEX - Nils Goroll Systemoptimierung

Schwanenwik 24
22087 Hamburg

Tel +49 40 2880 5731
Mob +49 176 636 90917
Fax +49 40 42949753

http://uplex.de
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.10 (SunOS)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iQIcBAEBCAAGBQJNYqnIAAoJEOUwvh9pJNURlyYQAJ7MR0BSv6dOnIKzu5T7Ig6r
p5NBn6eeO1o56bK8abOzK5nq/X+XZiIOUMDIeYQKaWnqKNcRjb3Y/iRz2+pSRBNa
HQAVQmX5S7TXQPpJQmxVR6LSKRsr5ch/i3JEVcI8tONSewKMF/Gh2KxEAMtCveLT
R2J7IaMr12JAroBpyCUgx8RsF9AqaNB3cHViGS2NjwpNLk26TxiyQ0qcRIyU4Jq7
r3e3AXy8pjK7csOseXw+bxcbiWGgCNIKW7gNwm+Xbr2DyiasJX2jhCKhN/VXcxtV
V0+KhEy7IUqKvXZoBIF45iOvTW9c2YRiV1hHqBbrOSwIeF1gJZk0Pje5icWECRFi
Bbd444OEGnmM6LrvMiWw9eXyzgERgUOF0SjVY3BrwYpRETT5abQQd1a9c2xIa2bt
CDALZABa+tOf09Y2uysfQ3H5MxqDiKlr8qwxmEBpAB/i8xueJBJQ/D0QG199n84P
fWUTo2OO+09ZJo6GcigwXDUYfE6gXF3x/MS5tPO30CxKJlxSvUfKmTYgaLkFMGlj
OOuKCOxdiYhZL71CjRhD6hzYuiYeSULYDLemqmliI5ckRYJceP7jcHDuR1y02Pyi
jsnbmWdqj5w6W54jFW7yyI+Xo6iONAnTTzPipHQKR8s4BXajt4BMj7TxJViIJ1pI
xgjq4ztSsm4NaNZraJhS
=RILD
-----END PGP SIGNATURE-----