best way to not cache large files?

[Ken Brownfield]

If you have control over the backend (Apache) it should be made to emit a
Cache-Control or Expires header to Varnish to make the object non-cacheable
*if the file is too large*.  Apache will know the file's size before a
request occurs.  I was talking about logic within Apache, not Varnish.  This
is how it's "supposed" to happen.

With Varnish, I see no way to avoid downloading the entire file every time.
 You can control whether the file *stays* in cache, but that's it.  If there
were a URL pattern (e.g., magic subdirectory), you could conceivably switch
to pipe in those cases.

Thinking out loud... HTTP servers will send a response to a HEAD request
with a Content-Length header that represents the length of the full object
had a GET been performed.

If your Apache does this (some configurations will disable this), one hack
would be to have Varnish send a HEAD request to Apache for every object, set
a req flag if the returned content length is too large, then restart, and
then have logic that will force pipe if it's too large, otherwise pass.
 This will double the hits to the back-end, however, so some conditionals
would help (only .mov, or only a certain subdirectory, etc.)  And I've never
tried changing a GET to a HEAD with VCL or inline-C.

But usually when something is that difficult, it's a square peg and a round
hole. :-)

FWIW,
-- 
kb



On Tue, Mar 15, 2011 at 16:56, Chris Hecker <checker at d6.com> wrote:

>
> I'm not sure I understand.  I have control over the back end, the front
> end, the middle end, all the ends.  However, I thought the problem was there
> was no way to get varnish to read the header without loading the file into
> the cache?  If that's not true, then shouldn't Content-Length be enough?
>
> Chris
>
> On 2011/03/15 13:16, Ken Brownfield wrote:
>
>> I'm assuming that in this case it's not possible for you to have the
>> backend server emit an appropriate Cache-Control or Expires header based
>> on the size of the file?  The server itself will know the file size
>> before transmission, and the reindeer caching games would not be
>> necessary. ;-)
>>
>> That's definitely the Right Way, but it would require control over the
>> backend, which is often not possible.  Apache unfortunately doesn't have
>> a built-in mechanism/module to emit a header based on file size, at
>> least that I can find. :-(
>> --
>> kb
>>
>>
>>
>> On Tue, Mar 15, 2011 at 00:42, Chris Hecker <checker at d6.com
>> <mailto:checker at d6.com>> wrote:
>>
>>
>>    Yeah, I think if I can't do it Right (which I define as checking the
>>    file size in the vcl), then I'm just going to make
>>    blah.com/uncached/* <http://blah.com/uncached/*> be uncached.  I
>>    don't want to transfer it once just to throw it away.
>>
>>    Chris
>>
>>
>>
>>    On 2011/03/15 00:40, Martin Boer wrote:
>>
>>        I've been reading this discussion and imho the most elegant way
>>        to do it
>>        is to have a upload directory X and 2 download directories Y and
>>        Z with
>>        a script in between that decides whether it's cacheable and move
>> the
>>        file to Y or uncacheable and put it in Z.
>>        All the other solutions mentioned in between are far more
>>        intelligent
>>        and much more likely to backfire in some way or another.
>>
>>        Just my 2 cents.
>>        Martin
>>
>>
>>        On 03/13/2011 05:28 AM, Chris Hecker wrote:
>>
>>
>>            I have a 400mb file that I just want apache to serve. What's
>>            the best
>>            way to do this? I can put it in a directory and tell varnish
>>            not to
>>            cache stuff that matches that dir, but I'd rather just make
>>            a general
>>            rule that varnish should ignore >=20mb files or whatever.
>>
>>            Thanks,
>>            Chris
>>
>>
>>
>>            _______________________________________________
>>            varnish-misc mailing list
>>            varnish-misc at varnish-cache.org
>>            <mailto:varnish-misc at varnish-cache.org>
>>
>>
>> http://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
>>
>>
>>
>>
>>
>>    _______________________________________________
>>    varnish-misc mailing list
>>    varnish-misc at varnish-cache.org <mailto:varnish-misc at varnish-cache.org>
>>
>>    http://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
>>
>>
>>
>>
>> _______________________________________________
>> varnish-misc mailing list
>> varnish-misc at varnish-cache.org
>> http://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/varnish-misc/attachments/20110315/e02c9bee/attachment-0003.html>