varnish fetching, stroring and serving cached 404 from backend server

[David Birdsong]

I've been struggling to understand why and how varnish is deciding to
cache a 404 response from a backend.  It's very tough to reproduce.
I've thrown millions of test URLs at varnish where my testing
application will respond back randomly with a 404 and I can't trigger
it.  Only user traffic is able to trigger this weird behavior.

Anybody have any experience with this?

Here's a varnishlog of a 404.
http://pastebin.com/386VU4rC

I'm working to capture the varnishlog of the resource actually getting
fetched and stored.