Varnish use for purely binary files

[Laurence Rowe]

2010/1/19 pub crawler <pubcrawler.com at gmail.com>:
> Wanted in inject another discussion heady item into this thread and
> see if the idea is confirmed in other folks current architecture.
> Sorry in advance for being verbose.
>
> Often web servers (my experience) are smaller servers, less RAM and
> fewer CPUs than the app servers and databases.  A typical webserver
> might be a 2GB or 4GB machine with a dual CPU.  But, the disk storage
> on any given webserver will far exceed the RAM in the machine. This
> means disk IO even when attempting to cache as much as possible in a
> webserver due to the limited RAM.
>
> In this "normal" web server size model, simply plugging a bigger RAM
> Varnish in upstream means less disk IO, faster web servers, less
> memory consumption managing threads, etc.  This is well proven basic
> Varnish adopter model.
>
> Here's a concept that is not specific to the type of data being stored
> in Varnish:
>
> With some additional hashing in the mix, you could limit your large
> Varnish cache server to the very high repetitively accessed items and
> use the hash to go to the backend webservers where ideally you hit a
> smaller Varnish instance with the item cached on the 2-4GB webserver
> downriver and have it talk to the webserver directly on localhost if
> it didn't have the data.

Given that you've already taken care of the common requests upstream,
you are unlikely to see much benefit from any form of caching -
performance will be determined by disk seek time. I suspect you would
see much more of a benefit in moving to SSDs for storage. Even cheap
MLC SSDs like Intel's X25-M will give great read performance.

Laurence