*.keep and if-modified-since

[Geoff Simmons]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

On 5/2/11 9:45 PM, Rich Rauenzahn wrote:
> I can't seem to get this to work (keeping a file around if its
> if-modified-since hasn't changed) -- I'm also having a hard time
> validating whether it is working or not.

Have you applied the patch that was sent up to varnish-dev? The latest
one can be found here:

http://www.varnish-cache.org/lists/pipermail/varnish-dev/2011-March/002842.html

This feature is not currently in the Varnish trunk, you have to apply
the patch. I've been working with the Varnish committers towards getting
it into a 3.0.x version, and parts of it are now in the trunk (the keep
attribute in VCL). The full functionality is not going to be in 3.0,
because the committers are not yet satisfied with the way it's
configured. As I've understood the discussion, it's mainly because the
feature adds a keep interval that is concurrent with the grace interval
for an object whose TTL has expired, and it adds some control over the
stale_obj for managing conditional requests, but there's nothing similar
for grace. Keep and grace could be handled more consistently, but would
require some thought and discussion, and the committers are focusing
their energy on getting 3.0 out the door.

Real Soon Now there should be an experimental branch in the trunk that
includes the IMS feature. It's on my to do list, as soon as I get some
customer work out of the way. (*grumble*)

> Right now I observe that the file is taking longer to download, so I
> assume it is pulling the entire file again from the apache backend.
> (I also see new entries in the apache log -- but I'm unclear if those
> would also occur with the if-modified-since header.)   I'm also
> unclear what to look for in the varnish log.

If you're using the patch, there are a few ways you can verify the IMS
feature:

- - varnishstat will show non-zero values for fetch_304, if Varnish has
sent conditional requests to the backend and received 304 responses; and
the patch adds the counter cond_not_validated, which counts the non-304
responses from the backend for conditional requests from Varnish.

- - If Varnish receives a 304 response from the backend, then it normally
sends the status "200 Ok Not Modified" back to the client. (Unless
Varnish decides for other reasons not to send a 200 status; for example,
if the client had sent a conditional request, then Varnish may respond
with 304 if the object qualifies.)

- - Apache logs show status 304 (field %s, which is in most of the
predefined custom formats). You could also use %{If-Modified-Since}i and
%{If-None-Match}i in your own custom log format to get the contents of
those headers.

- - Snoop the request/response traffic

> This is what I have right now for my vcl:
> 
> backend build_download1 {
>   .host = "build-download1";
>   .port = "80";
> }
> 
> sub vcl_recv {
>     #set req.ttl   = 1d;
>     #set req.grace = 1h;
>     #set req.keep  = 365d;
> }
> 
> sub vcl_fetch {
>     set beresp.ttl   = 1m; # lower to 1m for testing.
>     set beresp.grace = 1h;
>     set beresp.keep  = 365d;
> }
> 
> sub vcl_hit {
>     #set obj.ttl   = 1d;
>     #set obj.grace = 1h;
>     #set obj.keep  = 365d;
> }
> 
> I've looked at the diagram of the vcl subroutines, and I'm still
> having trouble figuring out where to put the timeouts.
> 
> What I want is for an object never to be thrown out if the file has
> not been modified on the backend (if it is removed, I want it to
> expire, obviously)

For that, you seem to be doing the right thing -- set beresp.keep in
vcl_fetch(). It's probably what most people will want to do.

req.keep in vcl_recv() sets an upper bound for the keep time just in the
current session (like req.ttl and req.grace). That is, if Varnish finds
a stale_obj with a keep time stored in cache that is greater than
req.keep, then Varnish will act as if req.keep is the keep time, but
won't change the keep time stored in cache.

You can set obj.keep in vcl_hit() if you want to change the keep time of
an object that is already in cache after it gets hit (also like obj.ttl
and obj.grace).

Please let us know how it goes -- any feedback will help to move the
feature along.


Thanks,
Geoff
- -- 
UPLEX Systemoptimierung
Schwanenwik 24
22087 Hamburg
http://uplex.de/
Mob: +49-176-63690917
-----BEGIN PGP SIGNATURE-----
Version: GnuPG/MacGPG2 v2.0.14 (Darwin)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iQIcBAEBCAAGBQJNvyVeAAoJEOUwvh9pJNURTUwP/1rYCzzFk9S0lK1sa+AIWJUa
g/DD1UROIfxZKgr0t33ooJjRcOt/xLzuTh3IwOoiodoPBPEljtS+V1+aKtQFeJei
p5AZDNVfefrbOTFrA9bP1w2S9Do1t/sMLxPBNTcV23MMWHPB4uolqr9YSC0FLYc+
JVLwDBCJnfidp89GXuIcL5LsPeZ3GlzSPdk38tMTFdvz3P72yewORmWfw3Wrt2o0
dtNMtXDNZ4w81tAAbgqGQehlD1edKyJszCXz55yb2Z3ZI6dqm7vJZmQ3n66lm5+K
1OR6ttujQqbQGn/74w5xxA4XbH1+MgNg8uUHOqAPuBg9MQhfw0FZsHQwnVloGx/4
/5ftdZt71sgAIkEA0sYUefwiaa3ctmWC7QYK2HWyKwRgP66N5JFEDBwQ3sQUAFBA
voEmIIceNeCYAPxLeCMauKciVv15XBPKSMz15+zt7nSuQ0khzTcXutsHkwqm0Cf7
wqZuvBUgJSJ1j2TqhH1o61WgVMUOATEeYJZg/vONfKIdNjyFTJ2bXayHicoYLZyI
N7XjtWWqJsfQJwKmQF31thDBhMy8LQRXRbrL/UlyMfAa61ULJEfOjhvaFJ40Jsez
Ls+IiJi/Il1zecsL7hpibkl6LDWtMoTQxq4MpJGcpea3bcIsu6eQ/pfQueNbOzj6
SqsH9O70EKRb+rSZBTPF
=FS/5
-----END PGP SIGNATURE-----