Streaming and backend conditional requests

Geoff Simmons geoff at
Thu Jan 12 12:32:02 CET 2012

Hash: SHA256

Hello all,

Yesterday there was a discussion about this subject on the
#varnish-hacking IRC channel, and we decided that a summary should be
sent up to varnish-dev, to ask for feedback. @phk, I hope we can get
your opinions.

For background on backend conditional requests (usually
If-Modified-Since), currently implemented in the branch experimental-ims
of the source repository, look here:

This implementation never chooses a busy object during hash lookup as
the stale object (the candidate object from cache to be validated with
the conditional request). The reason for this was mainly just to not
change too much about the way Varnish works. If a matching busy object
is found in HSH_Lookup(), then it returns just as it does in the current
master; a stale object is taken into consideration only if that doesn't

In 3.0 streaming, a streaming object always has the busy flag set; so a
streaming object can never become the stale object in experimental-ims.
As I've understood Martin, this will change in new streaming -- IIUC a
streaming object won't have the busy flag set, but will have a non-NULL

So something will have to change for IMS to work with new streaming. The
question, however, is whether passing over all currently streaming
objects as potential stale_objs for IMS is a good idea.

Suppose a very large object is streaming, which takes so long that its
TTL elapses before streaming is finished. One might say that this is a
poorly configured TTL, but it can nevertheless happen. When a new client
request for this object now arrives, it could conceivably be validated
against the backend as "Not Modified", so that it can be streamed from
Varnish to the second client as well, without having the backend send it
again. Reducing the bandwidth from backends for large objects is, after
all, a main goal of having conditional requests.

experimental-ims as currently implemented couldn't do that. If the
stale_obj is validated by the backend, then:

- - HTTP headers that the stale_obj has but the beresp object created for
the response doesn't have are filtered into the beresp object.

- - Storage for the stale_obj is duped into the storage for the beresp

"Dupe" is currently implemented just by copying the storage, although
the interfaces have been set up so that in the future, something more
efficient could be implemented -- what we've had in mind is that storage
can be shared with pointers by more than one object, and storage objects
have refcounts.

Last year, phk advised us to just implement the storage dupe with
copying for now, and implement something like refcounting in a future
version. As I understood it, this was so that we don't try to change too
many things at once; and I think it wasn't entirely clear how to
implement something like shared storage and refcounting for the
persistent stevedore (I've had a look and it hasn't been clear to me).

So as things are, experimental-ims couldn't try to validate an object
that is currently streaming, with or without the busy flag, because it
has to be able to copy its storage. The questions are:

- - Should backend validation stay this way, or should we try to change it
as suggested above (validate a currently streaming object)? (Opinions in
IRC were that this would definitely be worthwhile.)

- - If so, how should we manage the storage for the stale_obj and the
beresp object? Dupe by copying, as now implemented, would have to be

An idea on IRC was: rather than introducing refcounting into storage and
sharing it among objects, have the objects point to each other, and make
use of the refcounting for objects that already exists. So the beresp
object, with the updated headers, would have no storage of its own, but
would point to the stale_obj, indicating that its storage is to be found
there. I haven't fully seen through all of this (and may have restated
it incorrectly); but the idea of using a refcounting solution that has
existed and been tested for a long time seems promising.

Opinions, comments? What do you think?

- -- 
** * * UPLEX - Nils Goroll Systemoptimierung

Schwanenwik 24
22087 Hamburg

Tel +49 40 2880 5731
Mob +49 176 636 90917
Fax +49 40 42949753
Version: GnuPG v1.4.10 (SunOS)
Comment: Using GnuPG with Mozilla -


More information about the varnish-dev mailing list