varnish fetching, stroring and serving cached 404 from backend server

[Michael Loftis]

--On Wednesday, August 18, 2010 12:50 AM -0700 David Birdsong 
<david.birdsong at gmail.com> wrote:

> On Tue, Aug 17, 2010 at 7:59 PM, Michael Loftis <mloftis at wgops.com> wrote:
>> Varnish caching, or not caching, is completely controlled by the VCL.
>>  Not having yours noone can help you.  However, the *default* VCL
>> makes anything with a Cookie or HTTP Authentication header from the
>> client uncacheable.
>
> Fair enough, I should have known vcl would be helpful, *but I didn't
> think that statement was entirely true.
> "completely controlled by the VCL"...I thought there were boundaries
> that varnish operated in and no amount of vcl magic would override
> it...such as beresp.cacheable, if that boolean evaluated to false, I
> thought that varnish will never cache.  Is that incorrect?

<http://www.gossamer-threads.com/lists/varnish/misc/14286> i think that 
thread should answer your ?'s on beresp.cacheable, in short, it is not 
authoritative.  Further more 404's are perfectly cacheable.


You're caching all requests for a big list of extensions on line 91...

I'm not sure about case sensitivity in the req.http.cookie statement but 
since you're passing everything to lookup anyway in the end you *should* be 
caching 404s.  I'm actually reading this and trying to figure out why a 404 
*wouldn't* cache.  In fact the only places I don't see you caching are if 
it has an auth header, a cookie AND that cookie looks like one of your auth 
cookies.  That's the only reason I could see your code NOT caching. 
Expect, Auth headers, and a cookie that looks like your auth cookie.

So I'm guessing that once in a while a miss is resulting in a 404 which 
then (appropriately) gets cached.  If you don't want 404's getting cached 
set their TTL to 0 in vcl_fetch.  Just remember that'll leave your backends 
open to that sort of mischief.

It looks like tour vcl_fetch does do a fall through to the default 
vcl_fetch, so unless there are overriding expires or cache-control headers 
coming back on 404's, they'll be cached.  If you don't want them cached you 
need to pass on them in vcl_fetch.

>
> Anyway, here's the vcl:
> http://pastebin.com/egAfyT7U