Dealing with large unwanted objects

[Mark Moseley]

On Fri, Oct 29, 2010 at 9:14 AM, Taylan Develioglu
<tdevelioglu at ebuddy.com> wrote:
> I think what Per mentions can be solved by the pipe functionality.
>
> The problem is in knowing beforehand what size the object is. The only
> reliable way (especially with chunked-encoding) of knowing that is by
> fetching it first.
>
> Not much help, I know :)
>
>
>>
>> Just a comment to make everything crystal clear. I'm sorry if I'm
>> Captain Obvious here. When Varnish does a pass it will do so as a
>> store-and-forward kind of operation. So Varnish will turn to the
>> backend, fetch the object, free space in its store, fetch it deliver
>> it and then discard of it. The bad news is that it might knock out a
>> few object - the good news is that the space is quickly freed up
>> again.


Yup, that's what I was afraid of. 'pipe' doesn't work in vcl_fetch and
it'd be too late anyway. That's why I was curious if there was some
better way of accomplishing this that I'd missed. Since this is web
hosting, we've got no way to tell from the incoming request if
something's going to be big (besides file extensions at least). We
already route around varnishd if the request is for something that's
likely to be huge (.rar, .zip, .mpg, .wmv, etc), which is a bummer
since I'd love to cache smaller .rar/.zip/etc's, but without huge ones
wreaking havoc on the cache. It's good news that the space is freed up
quickly but when the object is half the size of the cache, it not only
knocks out a whole lot of objects but also puts a not-insignificant
strain on disks, for disk-based caches.

Maybe better to just consider this a suggestion for some future major
version, to have a hook that gets called after headers have been
fetched (or when the first chunk of returned data is seen that
contains the header break) but prior to vcl_fetch -- though this could
also be a huge performance hit to pause the retrieval.