Dealing with large unwanted objects

[Mark Moseley]

On Fri, Nov 5, 2010 at 1:10 AM, Tollef Fog Heen
<tfheen at varnish-software.com> wrote:
> ]] Mark Moseley
>
> | Wow, that's cool. I figured that the body had also been completely
> | fetched by the time vcl_fetch is invoked, so that changes things
> | completely. I'll definitely give this a try.
>
> It is in 2.0, but not in 2.1.
>
> | BTW, is there any "cast" trick to be able to use arithmetic operators
> | like ">" with beresp.http.Content-Length (or other numeric headers)?
>
> There'll be a way to do that in 3.0, yes.

So if it's not fetched the complete body yet, once 'restart' is
called, does varnish just dump the body? When I tried out your recipe
(which works like a charm, btw), I noticed that varnish still reads
the entirety of the first (i.e. the restarted one) request's response
body, before proceeding to fetch the restarted request. All this
happens over localhost, so it's quite fast, but in the interest of
efficiency, is there something I can set or call so that it closes
that first connection almost immediately? Having to refetch a 800meg
file off of NFS might hurt -- even if a good chunk of it is still in
the OS block cache.

To test, I set up Apache so that it'd return a 403 if the 'restart'
header is on the request, so that the first restarted request gets
fetched completely but the second one is basically empty. Given a 200
meg file, on a newly started varnish, the "SMA bytes allocated"
immediately jumped to 224265008. Once the first request finished, only
then did 'SMA bytes free' show up, also with 224265008. That would
make it appear that is storing the response. Or is it allocating the
memory in name only -- i.e. not actually doing anything -- and
possibly not causing varnish to knock things off the LRU?

BTW, this is 64-bit 2.1.4 on Debian Lenny with "-s malloc,4g".