Varnish caching issues, cache size, expires, cache resets, whatsgoing on?

[Brian Smith]

Demitrious Kelly wrote:
> On a happy note after tcmalloc varnish has stopped crashing every couple
> of hours.  We're just going to be hitting peak traffic for the week here
> in a bit, but so far so good.

How are things working out?

I read your blog post where you said requests go through pound, then 
varnish, then a custom server, then S3. In addition, you use memcached to 
cache some per-object metadata. Also, you don't cache things right away; you 
wait until an object has been requested several times before you let 
varnishd cache it.

Do you avoid caching responses right away due to the lack of an automatic 
LRU purging mechanism in varnishd? And, have you considered caching the 
responses in memcached directly instead of in varnishd? It seems to me that 
"varnishd -s malloc" and memcached are not too different.

Also, I am curious about why your pound server hashes on the host name only, 
instead of the whole URL.

See http://www.igvita.com/2008/02/11/nginx-and-memcached-a-400-boost/

Thanks,
Brian

>
> I took some graphs from munin and have made them available to look at if
> you are at all curious to look.
>
> http://blog.apokalyptik.com/wp-content/uploads/2008/05/mem.png
> http://blog.apokalyptik.com/wp-content/uploads/2008/05/load.png
> http://blog.apokalyptik.com/wp-content/uploads/2008/05/cpu.png