Varnish use for purely binary files

pub crawler at
Tue Jan 19 02:11:58 CET 2010

Wanted in inject another discussion heady item into this thread and
see if the idea is confirmed in other folks current architecture.
Sorry in advance for being verbose.

Often web servers (my experience) are smaller servers, less RAM and
fewer CPUs than the app servers and databases.  A typical webserver
might be a 2GB or 4GB machine with a dual CPU.  But, the disk storage
on any given webserver will far exceed the RAM in the machine. This
means disk IO even when attempting to cache as much as possible in a
webserver due to the limited RAM.

In this "normal" web server size model, simply plugging a bigger RAM
Varnish in upstream means less disk IO, faster web servers, less
memory consumption managing threads, etc.  This is well proven basic
Varnish adopter model.

Here's a concept that is not specific to the type of data being stored
in Varnish:

With some additional hashing in the mix, you could limit your large
Varnish cache server to the very high repetitively accessed items and
use the hash to go to the backend webservers where ideally you hit a
smaller Varnish instance with the item cached on the 2-4GB webserver
downriver and have it talk to the webserver directly on localhost if
it didn't have the data.

Anyone doing anything remotely like this?  Lots of big RAM
installations for Varnish.  I like the Google or mini Google model of
many smaller machines distributing the load.  Seem feasible?  2-4GB
machines are very affordable compared to the 16GB and above machines.
Certainly more collective horsepower with the individual smaller
servers - perhaps a better performance-per-watt also (another one of
my interests).

Thanks again everyone.  I enjoy hearing about all the creative ways
folks are using Varnish in their very different environments.  The
more scenarios for Varnish, the more adoption and ideally the more
resources and expertise that become available for future development.

There is some sort of pruning of the cache that is beyond me at the
moment to keep Varnish from being overpopulated with non used items
and similarly from wasting RAM on the webservers for such.

Simple concept and probably very typical.  Oh yeah, plus it scales
horizontally on lower cost dummy server nodes.

More information about the varnish-misc mailing list