Varnish crashing in production

[Barry Abrahamson]

> Within reason - there is no point in using a cache file that is many
> times the size of your data set.  I'm not sure how the allocator works
> in detail, but somewhere between 1x and 2x the size of your data set
> should be sufficient.

Varnish is sitting in front of about 600GB of data currently.  We add  
a header to files that were requested more than X times in Y seconds  
and tell varnish to cache anything with that header and don't cache  
rest.  When we were trying to cache everything we were running into  
serious IO issues (not to mention that we would run out of space  
eventually).   I am not sure how big that dataset is going to get.   
The Expires headers on the content is 1 week.  Once the crashing  
issue is resolved and varnish stays running for a while I will be  
able to get a better idea.

Thanks again,

Barry