Varnish as cache for image servers

[Anders Berg]

On Nov 19, 2006, at 18:23, Andrew Miehs wrote:

> Dear List,
>
> I only yesterday discovered Varnish. After spending most of today  
> reading and trawling the web on information concerning it, I am  
> dying to try it out on our web farm tomorrow.

Hi Andrew, and welcome to Varnish.

>
> My current problem is that I have 3 web servers, each with the same  
> 32GB of static image files. I am currently using Lighttpd running  
> on Linux 2.6 to deliver this static content to our customers. The  
> web servers are 32bit machines with 4GB of RAM. (Yes 64bit would be  
> better) :-). We currently have have approx 1000req/s per server  
> (files about 30k each). An F5 Loadbalancer is splitting the load  
> between these machines.
>
> # free
>              total       used       free     shared    buffers      
> cached
> Mem:       3930100    1917892    2012208          0     517032      
> 971216
> -/+ buffers/cache:     429644    3500456
> Swap:      1028152          0    1028152
> #
>
> The original idea was to let the linux kernel do all the caching  
> with its filesystem cache. Unfortunately, it does not seem to want  
> to use all the RAM available for this purpose. I was hoping that by  
> making more effective use of the available RAM, I would be able to  
> reduce IO Wait.
>
> I am aware that I can help do this with faster disks, bigger RAID,  
> etc, but I was hoping to try and improve the situation by using the  
> onboard RAM.
>
> Now for my questions:
>

I'll have to answer these questions based on my experience and maybe  
in a generic way.

> 1) Would you recommend using Varnish for this situation, or would  
> you suggest something like Apache with Mod_cache_memory or Zeus  
> with memory caching?

I would say that since your content is static I would in either go 2  
ways based on the traffic pattern and what your data is "doing". For  
example, where I work we have a rig that serv flash ads, and since  
some tags for ads today make a uniqe URL for each serv (even if  
content is satic). Since caching works bad in situations like this, I  
have ended up using a FreeBSD with accel filtering and apache, and  
that works fine. Mind that is with a small "often used" dataset.
But when it comes to "pure" static files I would say that Varnish in  
theory should be a perfect match for what you are trying to do (I say  
in theory since I have not used Varnish like that in a prod.  
evironment). The reason I say so is because Varnish uses some "state  
of the art" features in the OS (Even if they have been around a long  
time :) Like you say, let the OS do the stuff it knows how to do  
best.). Right now I use Varnish in a prod. environment with mixed  
content static/dynamic, and it works amazing. Read the attached PDF  
for a presentation and some stats.

I wanna add stuff here about 32 bit VS 64 bit, and talk some more  
about you dataset instead of answering those questions later. Since  
the 32 bit is slowly dying out we did not put much effort into it and  
you will see "stupid" limits like max size on filesize (cache size).  
We mmap the a file and let the OS do its work, and a 32 bit machine  
would, as you say, have a 3 GB limit (I have not tried Varnish on a  
32 bit machine, so I can't say where the limit is.). If that is  
"enuff" to make Varnish work at it's best for you depends on your  
dataset.  Lets say you only "often" serv 3 GB of that static content,  
well, then that is not a big setback, since Varnish will slowly  
change data out with the default LRU algorithm (we have planned more  
algorithms, and Varnish is made modular so it's easy to change to one  
you'd like/make). If you can cache the static content for lets say a  
week and new static content is added at a slow pace and not 30 GB a  
day, not much disk IO will get done I guess. I know people wanna use  
their hardware to the end, but Varnish running on 64 bit shines so  
good, in many cases it's an immediate return of investment buying new  
machines and running Varnish. I changed out 12 machines running Squid  
with 2 64 bit machines with Varnish :)

As I mentioned we wanna add more algorithms in the future, if there  
is need/requests for it, but right now we are gonna focues our work  
on fixing bugs, getting things to run rock-hard stable, and adding  
fetaures in VCL. We are also starting work with a spec for Varnish  
2.0 (ESI/ESI'ish behaviour beeing the most popular request feature it  
seems), and getting the governing board down. Our list of things we  
wanna see in Varnish is long and uber-cool. Stay tuned.

> 2) The ALPHA notes suggested that Varnish did not check for backend  
> errors - has this been fixed in the 1.0.2 version? IE: Can I use it  
> in a live situation and sleep at night?

We cut some corners before launch, and VCL (thats where backend  
loadbalancing and some other "save-backend" features are implemented)  
did not get as feature-rich as we hoped. But I am happy to inform  
that the break we took after the 1.0 launch, is now over and features  
like backend-loadbalancing and checking are high on the list and  
"quick" to implement I am told. So again, stay tuned. Also note that  
I use Varnish in prod., but note that I am using FreeBSD STABLE and  
Varnish works great with a sometimes overloaded backend.

>
> 3) Do you think running web server and Varnish on the same box  
> would cause a problem?

Have not tried it. But should work fine depending on RAM and  
loadtypes again. :)

> 4) How do you limit the size of Varnish? Is this limited by the  
> size of the -s file? or size that you set with -s? I realise that  
> this will be a max of 3GB with a 32bit kernel....

-s uses or makes a file depending on if there was a file there in the  
first place.

> 5) Is there a suggested 'startup command line' that I should use in  
> such a situation - like the one used for for your test run?
>
> and last but not least
>
> 6) As I am quite happy with the modified dates, etc that my web  
> server is delivering, do I need to use VCL scripts?

No, the default VCL will honor your own settings in most cases.



Hope that helps some.

I guess I should add that Varnish was my initative and I provided the  
funding for Varnish 1.0 and therefore I would have to be biased :)


YS
Anders Berg

>
>
> Thanks for getting through this mail!
>
>
> Cheers
>
>
> Andrew