Dealing with large unwanted objects

[Mark Moseley]

We've had varnish running in a web hosting environment for quite a
while. As such, most of the things we ask it to cache are of a sane
size, but being web hosting, customers come up with the most painful
corner cases. I've been looking for the "right" way to do this for a
few months now but I'm not even sure there is a way to do what I'm
looking for. What I'm trying to do is *not* cache an object that's way
too big (which will almost certainly knock other things out of the
working set). I'm doing a half-way job of it by calling 'pass' in
vcl_fetch, i.e.,

<snip>
sub vcl_fetch {
	if ( !beresp.cacheable ) {
		return ( pass );
	}

	if ( beresp.http.Set-Cookie ) {
		return ( pass );
	}

        # This is a header we add via apache if it knows some reason
not to cache, like IP restrictions.
	if ( beresp.http.X-VC == "no-cache" ) {
		return ( pass );
	}

        # This is the rule to knock out big files
	if ( beresp.http.Content-Length ~
"[0-9][0-9][0-9][0-9][0-9][0-9][0-9][0-9]" ) {
		return ( pass );
	}

	if ( beresp.status == 206 ) {
		return ( pass );
	}

	return ( deliver );
}
</snip>

So in this example, anything over 10meg gets 'pass'. The ">" operator
errors out here, btw, presumably because it's not cast to an integer.
The downside to doing this is that it seems to be too late to keep the
object from being put (temporarily) in the cache -- or at least the
memory appears to get fully allocated or if I'm using using a
disk-backed cache, it spins the disks wildly for a while if the object
is huge. And on our non-SSD boxes where we're using malloc, this can
easily exhaust varnish's memory and cause the child to die.

Ideally, as soon as it received the headers, I'd like to call 'pipe'
so varnish just streams the data through and doesn't disturb the
cache. Am I missing something terribly obvious or does such a phase
definitely not exist? And if not, is there a better way to do this?
I'm running mainly 2.1.3 on both i386 and 64-bit Debian Lenny, but
I've been tinkering with this problem since well back into 2.0.x.

Thanks!