Varnish as cache for image servers

[Andrew Miehs]

Dear List,

I only yesterday discovered Varnish. After spending most of today  
reading and trawling the web on information concerning it, I am dying  
to try it out on our web farm tomorrow.

My current problem is that I have 3 web servers, each with the same  
32GB of static image files. I am currently using Lighttpd running on  
Linux 2.6 to deliver this static content to our customers. The web  
servers are 32bit machines with 4GB of RAM. (Yes 64bit would be  
better) :-). We currently have have approx 1000req/s per server  
(files about 30k each). An F5 Loadbalancer is splitting the load  
between these machines.

# free
              total       used       free     shared    buffers      
cached
Mem:       3930100    1917892    2012208          0     517032      
971216
-/+ buffers/cache:     429644    3500456
Swap:      1028152          0    1028152
#

The original idea was to let the linux kernel do all the caching with  
its filesystem cache. Unfortunately, it does not seem to want to use  
all the RAM available for this purpose. I was hoping that by making  
more effective use of the available RAM, I would be able to reduce IO  
Wait.

I am aware that I can help do this with faster disks, bigger RAID,  
etc, but I was hoping to try and improve the situation by using the  
onboard RAM.

Now for my questions:

1) Would you recommend using Varnish for this situation, or would you  
suggest something like Apache with Mod_cache_memory or Zeus with  
memory caching?

2) The ALPHA notes suggested that Varnish did not check for backend  
errors - has this been fixed in the 1.0.2 version? IE: Can I use it  
in a live situation and sleep at night?

3) Do you think running web server and Varnish on the same box would  
cause a problem?

4) How do you limit the size of Varnish? Is this limited by the size  
of the -s file? or size that you set with -s? I realise that this  
will be a max of 3GB with a 32bit kernel....

5) Is there a suggested 'startup command line' that I should use in  
such a situation - like the one used for for your test run?

and last but not least

6) As I am quite happy with the modified dates, etc that my web  
server is delivering, do I need to use VCL scripts?


Thanks for getting through this mail!


Cheers


Andrew

---
Andrew Miehs